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REPORT  OF  PANEL  1 


TITLE:  Quality  and  Reliability  Management  Concepts 


OBJE 


§^6  fjKsU’^  / 

£^0  determine  those  objectives,  concepts,  policies  and  uniform 
procedures  essential  to  effective  quality  and. reliability  manage¬ 
ment  within  the  DoD.  (j 


TOPICS  DISCUSSED: 


1.  The  Problem  and  Its  Environment 


2.  Policy 

3.  Communication 

4.  Organization 

5.  Resources 

6.  Discipline 


7,  Integrated  Engineering 

8.  Incentive 
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THE  PROBLEM  AND  ITS  ENVIRONMENT 


1 .  Product  Envi ronmont  and  Performance 

The  problem  connected  with  quality  and  reliability  assurance 
is  inadequately  defined.  The  environment  in  which  the  problem  exists 
is  not  well  understood  by  most.  The  words  by  which  the  problem  is 
described  are  imprecise  and  lead  to  misxmderstanding.  There  is  a 
vague  feeling  that  quality  and  reliability  must  be  improved,  but  there 
is  no  current  measure  of  the  costs  for  such  improvement,  thus,  there 
can  be  no  understanding  of  the  trade-off  between  costs  for  quality- 
reliability  assurance  and  application  of  the  same  resources  to  the  basic 
product.  Clearly,  the  problem  must  be  much  more  clearly  defined. 

The  products  used  by  DoD  and  the  Federal  services  are,  in  general, 
good.  They  produce  the  required  results  in  the  hands  of  the  users.  The 
record  in  Vietnam  makes  it  quite  clear  that  our  airplanes,  our  guns,  our 
bombs,  and  our  supplies  are  adequate  for  the  task.  The  record  of  the 
Strategic  Air  Force  in  maintaining  its  deterrent  measures  with  B-47's, 
and  later  B-52's,  over  the  past  ten  years  is  spectacular  in  terms  of 
total  number  of  missions  accomplished,  low  abort  and  attrition  rates, 
and  high  credibility.  Outstanding,  too,  is  the  record  of  the  Polaris 
submarine  missile  system  which,  from  its  initial  deployment  in  Decem¬ 
ber  I960,  has  provided  the  nation  with  undetected,  moving,  deterrent 
bases  with  all  missiles  up  more  than  98  percent  of  the  time.  This 
record  is  more  outstanding  when  it  is  understood  that  the  first  stage  of 
this  missile  system,  i.  e.  ,  the  fleet  ballistic  submarine,  goes  at  least 
four  years  between  overhauls.  The  Minute  Mar.  and  other  missile  sys¬ 
tems,  land-based,  have  been  quite  impressive  in  their  availability  and 
accuracy. 

The  nations  of  the  world  are  beating  a  path  to  the  door  of  the 
United  States  for  acquisition  of  its  weapons  and  items  supporting  the 
weapons,  and  the  men  who  use  them.  This  is  not  just  because  of  the 
magnanimous  attitude  of  this  nation  in  providing  assistance  to  others, 
but  largely  because  there  is  a  technical  and  quality  superiority  in  these 
products. 

I 

At  the  same  time,  the  nation's  defense  services  continue  to  use 
products  which  were  built  during  the  Korean  War  episode  with  good 
results.  Indeed,  in  the  case  of  conventional  submarines/ Navy- support 
ships,  and  certain  other  equipments,  items  designed,  built  and  used  in 
World  War  II  are  still  being  utilized  with  a  high  degree  of  success. 

The  product  age  spectrum  must  be  considered. 
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a.  Product  Age  Spectrum 
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These  and  many  other  equipments  must  go  through  all  parts  of  the 
product  life  cycle  being  cared  for  under  the  widest  variation  of  climatic 
and  geographic  conditions  with  skilled,  semi-skilled,  and  unskilled 
labor  utilized  in  production,  storage,  maintenance,  use  and  final  dis¬ 
posal.  The  product  life  phase  spectrum  must  be  considered. 

b.  Product  Life  Phase  Spectrum 


It  is  also  clear  that  the  products  involved  range  from  simple  nuts 
and  bolts,  shoe  laces,  paper  supplies,  and  the  like,  through  increasing 
degrees  of  complexity  of  the  higher  performance  bombers,  fighter 
aircraft.,  missiles,  detection  systems  and  the  Polaris  submarines. 
Most  of  these  products  in  use  have  been  handled  by  the  complete  gamut 
of  unskilled,  semi-skilled  and  skilled  personnel  with  success.  The 
product  complexity  spectrum  must  be  considered. 
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c.  Product  Complexity  Spectrum 
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Of  great  importance  during  engineering  phases  of  product  life  are 
the  numbers  to  be  built  and  the  product  failure  consequence;  these 
spectra  are,  however,  closely  related  to  product  complexity  and  are, 
therefore,  included  in  consideration  of  that  spectrum. 

d.  Product  Numbers  (Inverse)  Spectrum 
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e.  Product  Failure  Consequence  Spectrum 
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Thus,  the  considerations  of  quality,  reliability  and  maintainability 
must  take  place  in  at  least  a  tridimensional  matrix  configuration;  first, 
in  the  widest  product  age  spectrum;  second,  through  the  widest  product 
life  phase  spectrum;  and  third,  in  the  widest  product  complexity  spec¬ 
trum. 

It  appears  unlikely  that  any  prescribed  single  policy  for  quality, 
reliability  or  maintainability  will  provide  a  satisfactory  solution  unless 
this  policy  is  grounded  in  the  common  sense  and  inherent  abilities  of 
men  in  the  system,  one  which  appeals  to  their  imaginations  and  exhorts 
from  them  their  basic  best;  a  "You  are  directed"  type  of  policy  will 
clearly  fail. 


Recommendation 

It  is  recommended  that  the  DoD  recognize  the  superior  performance 
of  its  products  and  provide  policy  for  the  future  which  encourages  all 
personnel  elements  involved  in  the  life  cycle  to  provide  best  effort. 

2.  Definitions  Required 

Inconsistent  definitions  of  quality,  quality  assurance,  reli¬ 
ability,  reliability  assurance  and  maintainability  have  been  provided 
by  the  DoD,  the  Service  Department  Washington  Headquarters,  service 
field  activities  and  subordinate  elements.  With  such  definitions,  con¬ 
tractors  and  other  producers  have  attempted  to  provide  the  best  possible 
product.  Each  contractor,  each  inspector,  and  each  government  rep¬ 
resentative  has,  of  necessity,  been  required  to  provide  his  own  inter¬ 
pretation  of  what  we  are  seeking  and  how  we  shall  obtain  it.  Under 
such  circumstances,  it  seems  very  understandable  that  we  have  con¬ 
fusion,  difficulty  and  many  different  sets  of  standards  for  the  same 
product  built  in  different  plants  or  for  the  same  product  ordered  by 
different  customers. 

Better  definition  of  the  words  used  would  considerably  improve 
the  probability  of  problem  solution. 

Of  equal  importance  is  the  recognition  that  quality,  reliability  and 
maintainability  are  closely  related  with  time  being  the  most  irrr  ortant 
variable  involved.  It  appears  quite  unlikely  that  QR  and  M  should  be 
brought  into  one  basic  product  contouring  discipline.  Since  Quality 
Assurance  and  Reliability  Assurance  are  also  involved,  a  "QARAM" 
discipline  would  offer  a  major  vehicle  for  management  of  the  problem. 


Roc  ommendation 


It  is  recommended  to  redefine  the  basic  words,  quality,  quality 
assurance,  reliability,  reliability  assurance  and  maintainability  in 
terms  that  apply  across  the  full  range  of  DoD  products;  it  is  suggested 
that  these  definitions  be  derived  sequentially  and  that  the  QARM  dis¬ 
cipline  be  accepted  as  the  management  vehicle  for  solution  of  this  pro¬ 
blem, 

3.  Need  for  Comprehensive  Directives 

Most  of  the  OSD  and  subordinate  general  directives  relating 
to  quality,  reliability,  maintainability  and  their  assurance  are  some¬ 
what  descriptive  of  mass  production  or  repeat  production  of  highly 
complex  or  high-value  items,  such  as  airplanes,  missiles,  etc.  These 
directives  are  usually  provided  by  persons  well  removed  from  the 
scene  of  production,  and  are  sometimes  couched  *n  words  which  have 
connotation  at  the  seat  of  the  Government  which  is  different  from  that 
in  use  by  producers  or  users.  Thus,  the  directives  generally  relate 
to  only  a  small  portion  of  the  total  product  spectrum  required  by  the 
DoD. 

Clearly,  there  is  a  need  for  directives  covering  the  entire  product 
spectrum  and,  if  necessary,  supporting  directives  covering  the  element* 
of  the  product  spectrum  of  particular  importance. 


Recommendation 

It  is  recommended  that  DoD  review  its  directives  to  better  define 
the  problem,  so  that  the  entire  spectrum  of  products  can  be  properly 
covered. 

4.  Need  to  Know  Costs 

There  exists  in  the  DoD  a  vague,  benign  desire  to  know  the 
costs  associated  with QARAM.  This  desire  is  apparently  associated 
with  the  feeling  that  higher  assurance  efforts  (and  their  costs)  will 
result  in  higher  quality,  reliability  and  maintainability  and,  thus,  a 
low  total  cost  during  the  life  of  the  product.  There  is  little  under¬ 
standing  or  discussion  of  the  trade-off  effects  of  putting  additional 
resources  into  the  production  effort  versus  the  assurance  effort.  There 
appears  to  be  insufficient  understanding  that  the  assurance  effort  does 
not  add  quality  to  the  product,  as  does  the  basic  production  effort. 

In  addition,  none  of  the  DoD  accounting  systems  or  information¬ 
gathering  systems  separate  out  those  operations  which  add  value  to  the 


product  from  those  which  merely  monitor  and  give  added  confidence 
that  the  product  does  meet  the  required  product  attributes  set  up  by  the 
customer. 

Contractors'  accounting  systems  similarly  do  not  often  separate 
these  costs  accurately.  While  there  should  not  be  a  requirement  that 
contractors  change  their  entire  system,  it  should  be  a  requirement 
that  QARAM  costs  can  be  derived  as  needed  by  the  Government. 

It,  therefore,  appears  that  there  can  be  little  opportunity  to  satisfy 
this  vague,  benign  desire  without  a  major  modification  of  basic  DoD 
information /accounting  structure  and  the  infusion  of  the  trade -off 
concept  as  an  important  consideration. 


Recommendation 

It  is  recommended  that  OSD  consider  modification  of  the 
accounting /information- gathering  structure  to  provide  cost  information 
which  differentiates  between  those  items  which  add  value  to  the  pro¬ 
duct  and  those  which  monitor  or  provide  confidence-level  information; 
that  the  DoD  require  trade-off  of  asset-utilization  between  value-adding 
operations  and  monitoring  operations  as  an  integral  part  of  its  cost 
considerations. 

5.  Distribution  of  Assets 

Nowhere  is  there  a  strong  statement  of  the  gains  to  be  made 
or  the  reason  for  application  of  more  assets  to  the  quality  and  reli¬ 
ability  problem  area.  Nowhere  is  there  a  discussion  of  the  time, 
valuable  skill  resource  conservation,  valuable  money  conservation, 
etc.  The  general  approach  that  things  will  be  better  if  we  have  more 
quality  and  reliability  assurance  pervades  all  of  the  discussions  and 
appears  to  be  the  basic  postulate  on  which  the  entire  problem  solution 
is  predicated  and  implementing  directives  generated. 

While  them  can  be  no  question  that  all  persons  involved  are  well 
motivated  in  a  aesire  to  improve  the  efficiency  of  our  product  life 
cycle,  there  does  not  exist  a  clear-cut  statement  of  thinking  which 
could  provide  the  basis  for  problem  solution.  This  is  partly  because 
insufficient  cost  data  exists,  insufficient  life  cycle  data  exists,  insuf¬ 
ficient  trade-off  information  for  asset  application  exists. 

There  is  considerable  doubt  as  to  the  value  of  requiring  more 
effort  be  diverted  from  product  generation  to  assurance  functions. 


8 


Nonetheless,  it  is  generally  accepted  that  significant  gains  can  be 
made  through  a  better  quaJity,  reliability,  and  maintainability  system. 

A  major  element  not  clearly  stated  in  current  DoD  problem  dis¬ 
cussion,  policy,  and  directives  is  the  increasing  rapidity  with  which 
technological  processes  make  obsolete  the  product  which  is  in  one  of 
the  generation  phases,  i.  e.  ,  concepts,  research  and  development, 
engineering,  procurement  or  prototype  testing.  Modern  technology  is 
moving  so  rapidly  that  the  old  in-series  mode  of  product  generation 
can  in  general  no  longer  be  accepted.  This  requires  major  parallel 
efforts  (concurrency)  with  a  high  guarantee  that  quality  and  reliability 
obtain  at  each  phase  of  the  product's  life,  particularly  during  the 
generation  portion  of  the  cycle.  This  requires  that  higher  percentages 
of  our  assets  must  be  placed  in  the  quality,  reliability  and  maintain¬ 
ability  effort  to  overcome  the  possibility  that  quality  deficiency  would 
prolong  the  product  generation  portion  of  the  cycle.  Thus,  quality, 
reliability  and  maintainability  techniques  offer  a  major  remedy  to  the 
potentially  serious  consequences  of  delays  which  could  render  a  pro¬ 
duct  obsolete  prior  to  its  introduction  for  use  in  the  field. 

Thus,  on  the  basis  of  efficiency  and  timeliness,  it  appears  that 
assets  should  be  diverted  to  increase  the  assurance  of  quality,  reli¬ 
ability  and  maintainability  in  new  products,  particularly  those  of  high 
complexity. 

The  NASA,  FAA,  GSA  and  NBS  participated  in  this  study.  The 
comments,  suggestions  and  recommendations  are  equally  applicable 
to  those  organizations.  The  reader  is  requested  to  mentally  add  those 
organizations  wherever  he  reads  "DoD"  and  to  add  their  headquarters 
wherever  he  reads  "OSD"  etc. 


Recommendation 

It  is  recommended  that  DoD  formulate  policy  to  guide,  for  new 
products,  the  diversion  of  assets  to  assurance  of  quality,  reliability 
and  maintainability. 
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POLICY 


1.  Definition  of  Problem  Areas  and  Solutions 

There  has  been  inadequate  definition  of  the  QARAM  problem, 
beginning  with  definitions  of  the  words  to  be  used  in  discussing  the 
problem.  Here,  quite  clearly,  is  the  requirement  for  a  major  policy 
statement  which  includes  not  only  the  definition  of  the  basic  words  to 
be  used,  but  also  indications  of  what  is  the  problem  which  is  being 
solved,  together  with  ideas  or  directives  as  to  how  it  should  be  solved. 

Several  documents  currently  exist  which  treat  a  portion  of  the 
problem.  They  are  partially  overlapping,  somev/hat  ambiguous,  and 
quite  often  contradictory  in  that  the  terms  and  the  connotations  of  the 
terms  are  not  the  same  in  each  of  these  documents.  There  is  a  need 
for  reducing  the  many  documents  to  a  few. 

Current  OSD  directives  on  policy  do  not  provide  a  sufficient  state¬ 
ment  as  to  the  role,  position,  resource  application,  costing,  or  other 
basic  elements  of  QARAM.  These  are  needed. 

Because  of  the  extremely  diverse,  tridimensional  nature  of  the 
QARAM  problem  within  the  DoD,  it  is  believed  that  there  cannot  be  a 
standardized  organization,  a  standardized  application  of  resources, 
or  a  standardized  single  policy  directive  which  would  be  specific 
enough  to  allow  the  individual  departments  or  their  field  activities  to 
operate  effectively. 

Therefore,  it  is  considered  that  the  DoD  policy  statement  required 
is  one  relating  to  the  functional  considerations  of  QARAM;  specifically, 
that  the  QARAM  elements  of  a  product  or  a  system  are  vital  and 
important  to  the  point  that  application  of  the  principles  involved  in 
QARAM  must  be  identifiable  and  auditable  from  the  conception  of  the 
idea  through  each  successive  element  of  the  product  producing  cycle. 
This  identification  and  auditability  must  be  defined  in  the  software  of 
the  product,  the  software  of  the  producing  systems,  the  organizational 
arrangements  of  the  producing  systems,  and  in  the  position  descrip¬ 
tions  of  certain  elements  within  each  of  the  producing  organizations. 
For  certain  highly  complex  products  or  systems  or  for  organizations 
whose  sole  mission  is  procurement  or  production  it  would  be  desirable 
to  specify  a  specific  QARAM  organization. 
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The  Government  role  in  weapons  systems  continues  to  increase  in 
relative  magnitude  as  the  complexity  of  the  products  increases.  In 
previous  wars  and  previous  preparations  for  wars,  elements  which 
were  in  use  in  the  community  effort  in  the  social  lives  of  individuals 
were  regularly  used- -knives,  guns,  bicycles,  automobiles,  trucks, 
ordinary  ships,  ordinary  boats,  ordinary  planes  were  used  in  World 
War  I.  With  increasing  sophistication  there  was  an  increasing  partic¬ 
ipation  of  Government  in  the  conceptual,  developmental,  producing,  and 
storage  phase,  even  prior  to  handing  the  item  to  a  military  user.  This 
increasing  role  has  accelerated  sharply,  after  World  War  II,  as  the 
frontiers  of  technology  have  been  pushed  outward  and  as  the  modes  and 
equipments  of  warfare  have  increased  well  beyond  the  frontiers  cur¬ 
rently  being  exploited  by  commercial  products. 

The  new  bombers  and  the  associated  systems,  the  new  missiles 
and  their  associated  systems,  new  submarines  and  their  associated 
systems,  all  are  well  beyond  those  items  which  are  in  current  use 
commercially  for  the  United  States  social  activities.  This  increasing 
phenomenon  results  in  increasing  requirements  for  the  Government 
to  better  define  its  role  in  the  development  of  equipments  and  facilities 
for  conducting  warfare. 

If  the  present  trend  is  continued  unchecked,  ultimately  the  Gov¬ 
ernment  will  provide  virtually  all  of  the  resources  and  all  of  the  direc¬ 
tion  as  to  how  these  products  will  be  developed,  produced  and  stored. 
The  Government  and  the  producing  activity,  whether  it  be  private  or 
Governmental,  would  essentially  have  to  develop  its  own  experience 
and  continue  to  develop  its  own  techniques.  Should  this  trend  continue, 
it  would  be  expected  that  the  statistics  of  use  which  corne  from  spread¬ 
ing  a  product  amongst  190  million  people,  the  experience  of  many  dif¬ 
ferent  techniques  and  technological  processes  being  brought  to  bear, 
the  experience  of  many  millions  of  use  incidents  would  largely  not  be 
available  or  vised.  There  is  a  large  additional  cost  factor  involved,  as 
well  as  the  possibility  of  not  being  able  to  develop  experience  fast 
enough  to  insure  a  satisfactory  product. 

On  the  other  hand,  products  such  as  guns,  automobiles,  commer¬ 
cial  aircraft,  etc.  ,  do,  by  exposure  to  the  entire  society  of  the  United 
States,  acquire  a  great  deal  of  statistical  experience  in  use.  Obviously, 
such  experience  and  such  knowledge  is  of  great  value  and  is  obtained 
essentially  as  a  by-product  of  today's  society.  It  appears  that  signif¬ 
icant  advantage  can  be  obtained  if  there  is  a  continual  effort  on  the  part 
of  Governmental  activities  engaged  in  conceiving  and  developing  new 
products  for  warfare,  if  these  products  can  be  pushed  from  the  Gov¬ 
ernmental  and  Government-captured  industries  into  the  commercial 
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industry.  It  appears  wise  that  a  policy  be  developed  which  indicates 
the  desire  of  the  OSD  to  continually  push  the  products  developed  for 
use  of  the  armed  services  in  the  direction  of  using  commercially 
developed  components,  elements,  sub-systems,  etc. 

The  continued  proliferation  of  splinter  engineering  discipline  or 
quasi-engineering  disciplines  from  the  basic  engineering  structure  of 
field  organizations  as  a  result  of  DoD  emphasis  is  cause  for  serious 
concern.  This  splintering  robs  the  engineering  effort  of  needed  talent, 
results  in  over-emphasis  of  the  splinter  area,  and  sometimes  degrada¬ 
tion  of  the  product.  While  there  can  be  no  question  but  that  certain 
techniques  from  time  to  time  need  additional  emphasis,  they  are  better 
served  when  they  are  provided  with  the  additional  emphasis  within  a 
basic  engineering  organization.  Since  the  OSD  has  no  basic  engineer¬ 
ing  organization,  it  would  appear  that  the  splinter  contouring  engineer¬ 
ing  disciplines  should  be  gathered  into  one  organization  or  at  least 
monitored  by  a  management  type  organization  to  determine  the  kinds 
and  amounts  of  resources  which  will  be  applied  in  each  discipline  and 
for  how  long.  Thus  the  concept  of  integrated  engineering  is  restated 
and  revitalized. 


Recomme  ndations 

It  is  recommended: 

1.  That  DoD  revise  and  reissue  its  basic  policy  documents  con¬ 
cerning  QARAM  to  improve  the  policy  directive  itself  and  to  provide  a 
firm  base  for  the  various  departmental  implementing  policy  directives. 

2.  That  the  OSD  policy  statement  on  QARAM  include  a  statement 
that  full  identification  and  auditability  for  QARAM  for  each  product  is 
required  from  conception  through  final  use  and  disposition  of  the  pro¬ 
duct,  such  identification  and  auditability  to  be  clear,  through  the  soft¬ 
ware  of  the  product,  the  product  itself,  the  software  of  the  producing 
organization,  the  organization  of  the  producing  activities,  and  in  the 
position  descriptions  of  personnel  involved  in  such  production. 

3.  That  OSD  include  in  its  basic  policy  a  statement  which  indi¬ 
cates  clearly  the  desire  to  use  commercially  developed  and  proven 
components,  equipments,  and  sub- systems  wherever  possible  and 
further,  to  seek  out,  where  possible,  substitution  of  commercially 
proven,  through  use,  elements  for  those  elements  of  systems  or  sub¬ 
systems  which  have  already  been  developed  through  the  Governmental- - 
private  industry  team  working  in  a  warfare  area. 
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4,  That  OSD  include  in  its  policy  statements  words  which  will 
re-emphasize  the  importance  of  integrated  engineering  vice  fractured 
engineering  which  comes  from  separate  application  of  splinter  disci¬ 
plines  to  the  product. 

2.  Definition  of  Government  Role 

There  are  statements  of  the  Government’s  role  in  QARAM 
which  contain  almost  all  elements  but  none  contain  all  of  them.  Many 
statements  contain-  only  a  few  elements.  It  is  important  that  the  total 
Government  role  be  properly  delineated  in  a  policy  document.  This 
total  role  involves  being  a  customer,  a  user,  and  a  monitor,  in  the 
broadest  sense  of  each  term.  As  a  customer  the  Government  con¬ 
ceives  the  basic  idea,  does  initial  engineering  to  determine  the  general 
description  of  a  product,  may  then  turn  over  the  production  of  the 
finalized  product  to  a  vendor  or  a  series  of  vendors,  or  may  do  the 
basic  engineering  itself  and  obtain  products  from  vendors  to  integrate 
into  the  larger  system.  As  a  user  the  Government  may  take  partially 
completed  items,  fully  completed  items,  tested  or  untested,,  use  them 
immediately,  store  them,  and  then  use  them,  restore  them,  and  reuse, 
repair,  maintain,  etc.  As  a  monitor  the  Government  must  monitor 
not  only  the  commercial  vendors  who  supply  products  to  go  into  these 
systems  but  also  must  monitor  its  own  efforts.  In  this  last  area,  moni¬ 
toring  its  own  efforts,  the  Government  has  not  been  as  diligent  as  in 
monitoring  the  efforts  of  the  vendors  which  supply  materials  to  it.  In 
order  to  be  successful  in  determining  communication  modes,  organi¬ 
zation  modes,  resource  allocations,  and  the  application  of  manage¬ 
ment  techniques  and  motivational  techniques,  to  this  problem  of  QARAM 
in  these  three  major  roles,  a  definitization  through  a  policy  document 
is  required. 

The  use  of  industrial  funding  concepts  in  certain  of  the  Govern¬ 
mental  producing  activities  has  resulted  in  managerial  techniques 
which  have  in  turn  provided  major  sources  of  cost  benefits  to  per¬ 
formance  type  budgets,  through  cost  analysis,  etc.  Naval  shipyards, 
for  example,  have  been  so  funded  for  the  past  fifteen  years.  Other 
Governmental  activities  have  been  industrially  funded.  It  appears 
that  extension  of  this  principle  of  industrial  funding  through  more  of 
the  Governmental  organizations  would  be  of  significant  advantage,  if 
accomplished  in  conjunction  with  the  relaxation  of  conflicting  admin¬ 
istrative  constraints. 

There  is  virtually  no  limit  to  industrial  funding  concept  appli¬ 
cations  if  the  basic  attitude  is  one  of  maximum  extension.  All  project- 
type  organizations  within  the  DoD  could  be  industrially  funded,  for 
exc  Triple;  the  Polaris  project,  the  Minute  Man  project,  the  ASW  project, 
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the  NIKE  ZEUS  project,  etc.  By  industrially  funding,  all  costs  related 
to  the  project  could  be  captured  in  one  convenient  accounting  system. 

The  accounting  system  could  further  be  sub -divided  into  as  many  ele¬ 
ments  as  desired  such  as  by  technological  specialization,  by  commod¬ 
ity,  by  time  of  application  of  resource,  etc.  By  extension  of  the 
industrial  funding  principle,  then,  costs  of  various  elements  of  projects, 
of  various  specialties,  of  various  disciplines,  could  be  reasonably  cap¬ 
tured. 

In  addition,  the  principle  of  industrial  funding  removes  from  Govern¬ 
mental  operation  many  of  the  inherent  problems  which  have  plagued 
effective  and  efficient  operation  in  the  past.  For  example,  where 
there  are  no  charges  connected  with  a  service  or  a  product,  those  who 
have  access  to  such  service  cr  product,  clamor  for  more  service,  for 
more  effective  service,  for  application  of  all  of  the  service  agencies 
resources,  etc.  This  is  true,  for  example,  in  certain  of  the  labora¬ 
tories;  and  it  is  true  of  various  service  organizations  which  are  sup¬ 
ported  from  the  DoD.  Again,  for  example,  the  DCAS  organization, 
which  services  the  Army-Navy-Air  Force,  and  other  Government 
activities,  is  not  paid  for  by  the  user  and  the  services  rendered  depend 
upon  the  goals  set  by  the  administration  of  DCAS  and  often  are  sub¬ 
stantially  biased  by  the  clamor  raised  by  the  using  agency.  Thus,  since 
there  are  no  costs  involved,  the  squeaking  wheel  gets  the  grease.  This 
appears  to  be  an  improper  way  to  acquire  good  service.  Further,  the 
activity  requesting  the  service  has  no  way  of  sending  additional 
resources  to  the  activity  providing  the  services.  The  activity  pro¬ 
viding  the  services,  under  current  concepts,  has  no  good  way  of 
acquiring  the  additional  resources  to  provide  the  services  which  are 
requested  by  the  using  organization.  Under  industrially  funded  con¬ 
cepts  of  providing  assets,  the  user  of  the  services  pays  for  those  ser¬ 
vices  which  he  requests  and  the  provider  of  the  service  is  able  to 
acquire  the  necessary  resources  to  provide  the  requested  services. 

His  performance  to  the  customer,  or  user  of  the  service,  io  that 
which  determines  repeats  and  additional  requests  for  the  service.  The 
manager  of  the  service  organization  is  able  to  attune  himself  to  the 
customer's  needs  rather  than  to  the  vagaries  of  an  organization  direc¬ 
tion  from  above  which  has  no  customer  response  bias.  Industrially 
funded  operations  almost  inevitably  operate  better  than  do  appropriation 
funded  activities  whose  loyalties  and  sources  of  funds  are  from  above. 
There  can  be  no  question  that  individual  activities  and  individual  persons 
find  ways  to  provide  services  under  current  allotment  support  circum¬ 
stances  and  some  of  the  services  are  reasonably  good.  Almost  with¬ 
out  exception,  however,  they  have  been  improved  when  the  industrially 
funded  concept  of  Government  operation  is  invoked. 
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Field  activities  do  not  have  a  good  way  of  providing  career  patterns 
for  QARAM  personnel  nor  a  good  way  to  move  them  about  from  one 
position  to  another  in  slightly  different  disciplines.  Further,  manage¬ 
ment  has  few  incentives  to  eliminate  persons  or  functions  not  being 
fully  utilized  because  of  known  difficulty  in  reacquiring  assets.  Were 
these  problems  to  be  solved,  considerable  increase  in  effectiveness 
would  result.  There  could  be,  for  example,  ladder  type  careers  for 
QARAM  personnel  from  field  activities  to  headquarters  to  other  field 
activities,  etc.  ;  incentive  is  required  for  management,  for  quality 
personnel  and  for  the  organization  in  general. 

Government  buying  activities  have  a  responsibility  for  the  adequacy 
of  product  and  are  the  recipients  of  user  complaints  with  respect  to 
product  deficiencies.  Representatives  of  buying  activities  have  a  legit¬ 
imate  purpose  in  visiting  vendors  plants.  This  legitimate  purpose  is 
their  concern  with  product  adequacy.  Their  visits  must  be  coordinated 
with  the  contract  administering  sigency  and  their  findings  must  relate 
to  adequacy  of  product.  They  should  not  duplicate  the  responsibility  of 
the  contract  administering  activity  for  determining  and  approving  vendor 
QARAM  systems  and  implementation  thereof.  Their  findings  relative 
to  hardware  deficiencies  may  well  be  traceable  to  QARAM  system  defi¬ 
ciency;  however,  all  such  comments  made  must  stem  from  the  hard¬ 
ware  deficiency  and  not  from  primary  duplicative  review  of  QARAM 
systems  and  procedures. 

The  problem  becomes  one  of  even  greater  magnitude  when  Gov¬ 
ernment  agencies  outside  DoD  place  requirements  on  the  same  vendors 
and  find  that  there  is  a  definite  need  to  frequent  the  same  vendors  plant 
to  ensure  that  the  QARAM  objectives  are  achieved.  To  reduce  this 
need  for  overlapping  of  resources  and  reduce  total  Government  oper¬ 
ating  costs,  there  should  be  a  joint  DoD  and  independent  agency  QARAM 
functionary  that  will  provide  the  level  of  confidence  and  acceptance  in 
the  DoD  field  activities  efforts. 

In  a  diverse  spectrum  of  products  such  as  that  monitored  by  the 
DoD,  there  can  be  no  set  pattern  of  operation  involving  standardized 
organization,  standardized  equipment,  standardized  applications  of 
resources,  etc.  There  can,  however,  be  applications  of  the  basic 
principles  by  which  men  live  and  acquire  superior  performance  attrib¬ 
utes.  One  of  these  principles  is  honesty.  This  matter  is  generally 
handled  in  a  straightforward  way  and  is  well  understood.  Another 
important  principle  is  that  of  incentive,  which  although  equally  impor¬ 
tant  with  honesty,  is  often  not  considered  and  when  considered,  is 
often  not  applied  well.  Yet  incentive  is  the  principle  which  causes  men 
to  do  more  than  merely  exist  and  results  in  the  maximum  progression 
of  the  human  race. 
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OSD  and  subordinate  QARAM  directives  are  not  couched  in  incen¬ 
tive  terms,  often  they  result  in  dis- incentive.  The  directives  often, 
as  the  name  implies,  "direct"  individuals  to  perform  in  a  certain 
manner.  Direction  of  this  nature  is  satisfactory  in  times  of  extremes, 
in  times  of  emergency,  in  certain  warfare  conditions.  However,  this 
does  not  result  in  maximum  application  of  human  endeavor.  On  the 
other  hand,  provision  of  incentive  almost  always  encourages  maximum 
human  endeavor  and  maximum  progression  within  the  shortest  possible 
time. 

Incentive,  then,  and  application  of  incentive  ideas  should  be  a 
major  element  of  OSD  policy  statements. 


Rec  ommendations 


It  is  recommended: 

1.  That  the  Government  role  as  a  customer,  user  and  monitor  in 
QARAM  be  defined  in  a  policy  document  issued  by  the  OSD. 

2.  That  OSD  include  in  its  policy  statements  that  maximum  use 
will  be  made  of  industrial  funding  in  its  subordinate  activities  involved 
in  product  generation  and  production;  this  applies  particularly  to  DCAS. 
Such  use  should  be  in  conjunction  with  the  relaxation  of  conflicting 
administrative  constraints. 

3.  That  OSD  study  types  of  incentive  to  be  applied  to  add  organ¬ 
izations  to  provide  management  and  QARAM  personnel  with  ways  of 
decreasing  costs  and  obtaining  gains  themselves  thereby. 

4.  That  OSD  clearly  define  and  delineate  the  interface  between 
buying  activities,  contract  administration  activities,  and  vendors  and 
provide  for  the  enforcement  of  the  delineation  defined. 

5.  That  OSD  take  action  to  expand  the  existing  NASA/DoD  Reli- 
Ability  and  Quality  Assurance  Committee  to  include  the  other  interested 
Government  agencies  as  fuil  members  which  will  provide  for  face-to- 
face  discussion  and  resolution  of  common  problems  and  establish  an 
objective  to  eliminate  unnecessary  duplicative  activities. 

6.  That  DoD's  QARAM  policy  directives  contain,  without  excep¬ 
tion,  an  element  of  incentive  and  that  no  directive  be  issued  which  is 
not  basically  incentive  in  nature. 
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3.  Definition  of  Vendor's  Role 


The  role  of  the  vendor  or  equipment  producer  in  the  DoD 
scheme  is  not  sufficiently  defined  for  effective  QARAM  application. 
While  the  policy  clearly  indicates  that  the  vendor  must  provide  a 
hardware  product  which  is  satisfactory,  there  are  other  elements 
which  may  be  equally  important.  These  generally  must  be  obtained 
from  the  vendor  through  the  application  of  profit  or  other  incentives. 
For  example,  better  ideas  on  how  to  make  the  product  are  desired; 
ideas  on  how  to  reduce  costs  of  the  product;  ideas  on  how  to  improve 
effectiveness  of  the  product,  or  how  to  substitute  a  commercially 
available  element  for  a  specifically  designed  element.  These  kinds 
of  ideas  need  to  be  extracted  from  the  vendor  as  part  of  the  normal 
way  of  doing  business.  The  current  OSD  directives  regarding  the 
vendor's  role  and  the  incentives  to  the  vendor  in  doing  his  portion  of 
the  job  are  not  explicit,  except  in  a  few  specific  areas,  such  as  value 
engineering.  This  lack  of  incentive  description  in  DoD  policy  direc¬ 
tives  makes  it  difficult  for  vendors  to  forward  unusual  ideas  or  con¬ 
cepts  which  would  have  the  effect  of  improving  the  product,  making 
it  more  effective  and  less  costly.  Generally,  contracting  agencies 
of  the  Government  will  not  allow  the  vendor  any  profit  on  an  item 
which  has  been  reduced  in  cost,  unless  such  reduction  comes  within 
one  of  the  approved  incentive  plans.  There  needs  to  be,  however,  an 
opening  of  the  dam  in  this  regard  so  that  vendors  may  forward  ideas 
which  result  in  a  better  product  without  regard  to  contract  incentive 
terms  with  the  expectations  that  such  suggestions,  if  adopted,  will 
result  in  negotiated  cost  reductions  of  the  product  in  which  the  vendor 
would  receive  a  portion  up  to  fifty  percent.  It  appears  that  OSD  policy 
statements  defining  the  role  of  vendors  should  include  words  which 
provide  the  vendor  with  incentive  to  supply  information  as  to  better, 
quicker,  more  efficient,  more  effective  ways  of  performing  the  same 
function. 


Recommendation 

I*  is  recommended  that  the  OSD  policy  statements  include  words 
which  would  have  the  effect  of  providing  incentive  to  vendors  to  do 
more  than  just  provide  the  hardware  requested,  but  provide  also 
suggestions  as  to  better  ways  to  do  the  job. 
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COMMUNICATION 


1 .  Directives 

QARAM,  as  used  in  this  report,  is  an  abbreviation  for  the 
general  areas  of  quality,  quality  assurance,  reliability,  reliability 
assurance  and  maintainability. 

Basic  QARAM  terms  are  not  defined  consistently.  Efficient  man¬ 
agement  of  the  quality  and  reliability  function  within  the  DoD  is  difficult 
because  different  definitions  for  the  same  fundamental  quality  and  reli¬ 
ability  terms  exist  simultaneously  in  basic  directives  used  by  various 
field  activities. 

This  compounding  of  the  basic  communication  problem  leads  to 
unnecessary  lack  of  uniformity  of  operation,  to  inconsistencies  in 
policy,  to  unnecessary  variations  in  organization  and  to  general  con¬ 
fusion. 

Some  examples  of  differences  in  definitions  found  by  Panel  1  in  a 
review  of  various  directives  are  given  in  the  following: 

a.  Quality  Assurance.  "Quality  Assurance  comprises  a 
planned  and  systematic  pattern  of  all  actions  necessary  to  provide  ade¬ 
quate  confidence  that  the  product  will  perform  satisfactorily  in  service. 

(From  MIL-STD-109A  30  Oct  1961) 

"A  planned  and  systematic  pattern  of  all  actions  necessary  to  pro¬ 
vide  confidence  that  material  conforms  to  established  technical  require 
ments  and  achieves  satisfactory  performance  in  service." 

(From  DoD  Directive  4155.  11  of  17  June  1965) 

"A  planned  and  systematic  pattern  of  all  actions  necessary  to  pro¬ 
vide  adequate  confidence  that  the  end  item  will  perform  satisfactorily 
in  actual  operation.  " 

(From  NASA  NHB  5330.  7  Apr  1966  Edition  and  HNDBK  for  Q  &  RA 

Procedures  and  Standards  June  1965) 

"QA  is  a  planned  and  systematic  pattern  of  review,  audit,  and 
analysis  actions  necessary  to  provide  adequate  confidence  that  the  prod 
uct  conforms  to  the  requirements  specified  by  the  customer.  Audits  of 
processes  and  products  to  assure  management  th*  ♦.  controls  ,re  in  fact 
being  carried  out  and  results  are  within  control  limits,  are  a  part  of 
Quality  Assurance." 

(From  PTSMH  NAVSH1PYD  INST  4855.  3A  of  16  Mar  1965) 
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b.  Quality  Control.  "A  management  function  whereby  control 
of  quality  of  raw  or  produced  material  is  exercised  for  the  purpose  of 
preventing  production  of  defective  material.  " 

(From  MIL -STD - 1 09A  30  Oct  1961) 

"A  management  function  to  control  the  quality  of  articles  to  con¬ 
form  to  quality  standards.  " 

(From  NASA  NHB  5330.  7  Apr  1966  Edition,  Appx  G  and  HNDBK 

for  Q  &  RA  procedures  and  stds  June  1965) 

c.  Quality.  "The  composite  of  material  attributes,  including 
performance.  " 

(From  DoD  Directive  4155.  11  of  17  June  1965) 

"A  matter  of  conformance  of  product  to  explicitly  defined  require¬ 
ment,  i.  e.  ,  the  goodness  of  a  manufactured  item  when  judged  in  rela¬ 
tion  to  the  procurement  package.  " 

(From  MUCOM  Pamphlet  700-1  of  9  July  1965) 

d.  Reliability.  "Probability  that  material  will  perform  its 
intended  function  for  a  specified  period  of  time  under  stated  conditions.  " 

(From  DoD  Directive  4155.  11  of  17  June  1965) 

"Reliability  is  the  ability  of  a  product  to  perform  its  intended  func¬ 
tion  throughout  its  predetermined  life  span,  under  operating  conditions 
for  which  it  was  designed.  " 

(From  NHDBK  for  Q  &  RA  Procedures  &  Stds  June  1965) 

"The  inherent  capability  of  a  product  (of  given  design)  to  meet  the 
performance  requirements  of  the  user  for  the  period  of  time  specified, 
i  e.  ,  to  perform  as  intended  at  desired  time  under  desired  conditions. 
The  reliability  of  an  item  is  generally  stated  in  terms  of  probability  of 
success  under  a  given  set  of  conditions.  " 

(From  MUCOM  Pamphlet  700  1  of  9  July  1965) 

"The  probability  that  a  system,  sub-system,  component  or  part 
will  perform  its  required  functions  under  defined  conditions  at  a  desig¬ 
nated  time  and  for  a  specified  operating  period.  " 

(From  NASA  NHB  5330.  7,  Appx  G  Apr  1966  Edition) 

'The  probability  that  material  will  perform  its  intended  function  for 
a  specified  period  under  stated  conditions." 

(From  M1L-STD -72 1 A  2  Aug  1962) 
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"The  probability  that  systems  or  components  will  perform  their 
intended  function  for  a  specified  period  under  stated  conditions.  " 

(From  NAVSHIPS  0900-002-3000) 

e.  Maintainability.  "The  probability  (when  maintenance 
action  is  initiated  under  stated  conditions)  of  restoring  a  system  to  its 
operational  conditions  within  a  specified  total  down  time.  " 

(From  MIL-STD-721) 

"The  speed  or  economy  with  which  a  system  or  component  can  be 
kept  in,  and/or  restored  to  full  performance  capability.  " 

(From  NAVSHIPS  0900-002-3000) 

f.  Inspection.  "The  examination  (including  testing)  of  supplies 
and  services  (including,  when  applicable,  raw  materials,  documents, 
data,  components  and  intermediate  assemblies)  to  determine  whether 

the  supplies  and  services  conform  to  technical  requirements." 

(F  rom  MIL-STD-109A  and  DoD  Directive  4155.  11  of  11  June  1965) 

"The  examination,  including  testing,  of  contract  work,  articles  and 
services  to  detei  nine  conformance  to  contract  requirements.  " 

(From  ONM  INST  5000.  3  Vol  2  of  5  July  1963  Chap  7) 

2.  Recommended  New  QARAM  Definitions 

Panel  1,  after  careful  consideration  of  all  material  reviewed, 
offers  for  consideration  the  following  definitions: 

a.  Quality  Assurance.  "A  planned  and  systematic  pattern  of 
actions,  audits  and  analyses  necessary  to  provide  adequate  confidence 
that  product  attributes  conform  .o  specified  requirements.  " 

b.  Quality.  "A  condition  achieved  by  meeting  all  specified 
product  attribute  requirements.  " 

o.  Product  Attribute.  "A  visible  or  meaeureable  character¬ 
istic  or  condition  that  is  capable  of  being  verified  at  a  single  point  in 
time.  The  totality  of  product  attributes  must  be  determined  and  estab¬ 
lished  to  meet  the  intended  end  use  of  the  product.  " 

d.  Quality  Control.  "A  planned  and  systematic  pattern  of  those 
actions  necessary  for  the  control  of  product  attributes  and  the  processes 
that  affect  them.  " 

e.  Reliability.  "The  continuing  conformity,  over  time,  of 
product  attributes,  to  established  requirements,  expressed  as  a  prob¬ 
ability.  " 
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f.  Reliability  Assurance.  "A  pattern  of  actions,  audits  and 
analyses  necessary  to  provide  adequate  confidence  that  reliability 
requirements  prescribed  for  the  product  are  in  fact  being  met.  " 

g.  Maintainability.  "The  restorability  of  product  attributes 
within  the  specified  expenditure  of  resources,  (time,  money,  manpower, 
materials  and  facilities.  )  expressed  as  a  probability.  " 


Recommendation 

It  is  recommended  that  the  OSD  take  appropriate  steps  to  insure 
that  the  definitions  used  for  basic  QARAM  terms  are  standard  in  all  of 
its  organizations.  Further,  that  those  definitions  provided  by  Panel  1 
in  the  foregoing  be  considered  in  determining  what  the  standard  defini¬ 
tions  shall  be. 

3.  QARAM  Discipline 

Many  things  must  occur  between  the  formulation  of  a  new  idea 
and  the  delivery  of  a  satisfactory  end  product  to  the  customer.  For  one, 
product  attributes  must  be  determined  by  the  application  of  basic  engi¬ 
neering  concepts.  For  another,  product  attributes  must  be  modified 
to  suit  customer  requirements  for  safety,  reliability,  maintainability 
and  so  forth  by  the  application  of  these  contouring  disciplines. 

The  QARAM  discipline  involves  the  integration  and  comingling  of 
basic  engineering  and  the  contouring  disciplines,  starting  at  the  begining 
of  the  life  cycle,  so  that  necessary  interplay  and  trade-off  takes  place 
while  the  basic  product  is  being  engineered  and  developed.  This  concept 
recognizes  quality  as  an  immediate,  measurable  condition,  achieved  as 
product  attribute  requirements  are  met,  and  reliability  as  the  time 
extension  of  quality- -a  reducing  probability  that  product  attribute 
requirements  will  stay  satisfied  over  a  period  of  time.  Maintainability 
then  becomes  a  measure  of  the  restorability  of  product  attributes  to  a 
specified  quality  level  under  stated  conditions,  expressed  as  a  probability. 

Because  produce  attributes  are  so  interrelated,  optimization  of 
cost,  quality  and  operational  requirements  demands  that  they  all  be 
considered  together  in  an  integrated  engineering  environment  which 
includes  the  QARAM  discipline. 

The  DoD  and  implementing  military  department  directives  in  QARAM 
are  verbose,  ambiguous,  blurred,  extremely  extensive  and  duplicative, 
yet  omissive  and  defective.  There  are  many  hundreds  of  thousands  of 
pages  written  to  cover  the  area.  There  is  no  question  that  product 
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failure  has  great  impact  in  critical  military  or  civilian  applications  and 
every  service  element  seeks  to  establish  the  best  possible  safeguards 
through  definitive  specifications  and  written  operation  procedures. 

While  applicable  to  the  instant  problem,  these  are  generally  not  accept¬ 
able  elsewhere  because  their  purpose  is  not  understood.  Further,  the 
very  extensive  nature  of  these  communications  makes  it  unlikely  that 
even  the  most  diligent  of  contracting  activities  or  contractors  could 
carry  them  out.  It  is  doubtful  that  is  is  possible  to  understand  them. 

The  proliferation  of  the  same  idea  in  many  different  ways  as  a  result 
of  expanding  directives,  or  the  description  of  the  same  idea  from 
different  elements  within  the  DoD  or  the  military  establishments, 
results  in  an  almost  hopeless  situation  at  the  product  producing  level. 

The  problem  is  similar  to  that  which  existed  in  the  field  of  procure¬ 
ment  before  ASPR.  There  is  a  great  need,  therefore,  for  reducing 
the  number  of  directives,  for  making  them  briefer,  clearer,  to  improve 
the  communication  channels.  DoD  needs  a  group  of  technical  specialists 
to  coordinate,  draft,  promulgate  and  keep  up  to  date  one  set  of  clearly 
written  directives  on  QARAM. 

There  does  not  exist  today  a  good  QARAM  information  feedback 
system  which  provides  to  all  elements  of  supervision  in  the  Govern¬ 
mental  chain  an  indication  of  the  efficiency  and  effectiveness  of  the 
control  disciplines  which  have  been  directed.  Communication  currently 
is  a  one  way  street  from  the  top  down,  with  no  effective  information 
feedback  system  required  or  provided  through  the  same  channels. 

Many  other  information  feedback  systems  do  exist,  however,  for 
example,  through  accounting  channels,  through  defectives  reports, 
through  special  surveys  which  are  continually  being  ordered--none  of 
which  provide  a  full  picture  of  the  situation  as  regards  QARAM. 

Effective  communication  requires  a  feedback  system  which  provides 
the  essential  elements  of  the  discipline. 

To  insure  feedback  requires  engineering  assistance  which  is  effec¬ 
tive  in  two  directions.  This  is  a  particularly  important  requirement  in 
the  QARAM  area. 


Recommendations 


It  is  recommended: 
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1.  That  OSD  institute  a  directive  review  in  the  areas  of  QARAM 
which  would  result  in  fewer,  briefer,  and  better  directives.  Along 
v/ith  such  review  should  be  a  plan  to  carefully  screen  new  directives 
and  produce  them  to  the  same  condition  of  clarity  as  is  provided  by  the 
basic  recommendation  above. 

2.  That  OSD  establish  in  cooperation  with  other  Government 
agencies  a  project  group  of  high  level  specialists  to  accomplish  the 
required  review  and  to  reform  all  QARAM  and  related  specifications 
and  instructions  into  one  universal  set  of  directives. 

3.  That  the  OSD  study  the  need  for  an  effective  feedback  system 
in  the  QARAM  discipline  and  if  a  determination  is  made  that  such  is 
required,  that  a  group  be  convened  to  establish  the  mode  of  this  effec¬ 
tive  communication. 

4.  Collation  in  QARAM  Handbook 

A  comprehensive,  administrative  handbook  on  QARAM  is 
needed.  It  should  include  a  complete  index  of  QARAM  directives  and 
should  be  compiled  and  edited  by  top  peode. 

The  quality  of  material  furnished  to  the  Armed  Services  is  the 
result  of  teamwork  between  industry  and  the  responsible  Government 
agency.  Quality  of  the  product  produced  is  a  function  of  the  design, 
the  manufacturing  processes  and  controls,  the  ability  to  maintain  under 
use  conditions  and  to  operate  under  adverse  conditions. 

Total  quality  requires  an  integrated  effort  from  the  interface  of  all 
the  associated  disciplines  such  as  QARAM,  human  factors  engineering, 
maintenance  engineering,  safety  and  logistic  support. 

A  DoD  level  guidance  document  of  the  handbook  type,  integrating 
the  related  QARAM  oriented  disciplines,  would  serve  as  a  useful  tool 
in  the  management  of  many  DoD  programs.  Currently,  industrial  firms 
may  be  doing  business  with  more  than  one  military  service  or  with  FAA, 
NASA,  or  GSA.  It  appears  advantageous  to  standardize  administrative 
procedures  for  QARAM  and  other  interface  disciplines  and  techniques 
requiring  participation  by  the  contractor. 

Certain  differences  will  always  occur  when  dealing  with  Army, 

Navy,  or  Air  Force,  but  on  the  whole,  many  of  these  differences 
are  without  just  cause. 

Product  vendors  have  recognized  the  interface  problems  associated 
with  QARAM  related  functions,  and  many  leading  vendors  manage  them 
for  their  relatively  small  range  of  products  with  "product  assurance" 
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elements.  Integration  of  these  functions  permits  increased  emphasis 
from  management  and  also  encourages  a  combining  and  simplification 
of  policy  directives,  instructions  and  definitions. 

Although  the  DoD  problem  is  several  orders  of  magnitude  more 
complex  than  that  of  a  product  vendor  or  system  vendor,  it  is  apparent 
that  a  DoD  level  handbook  covering  the  integrated  functions  associated 
with  these  closely  related  disciplines  would  have  many  advantages. 
General  procedures  and  techniques  necessary  to  influence  QARAM 
programs  could  be  supplemented  with  certain  elements  of  human 
factors  engineering,  safety  and  logistics  programs. 

There  exists  today  no  good,  complete  listing  of  the  applicable  DoD 
or  service  directives  which  are  oriented  towards  QARAM.  These 
directives  are  issued  in  many  ways --as  DoD  directives,  military 
department  directives,  field  directives,  as  handbooks,  instructions, 
military  specifications,  boiler  plate  applications  to  purchase  contracts, 
specifications  in  purchase  contracts,  interpretations  as  a  result  of 
discussion  of  the  provision  of  inspection  requirements  in  contracts,  etc. 

It  is  quite  clear  that  even  the  elementary  listings  which  have  been 
attempted,  including  a  list  of  QARAM  directives  and  specifications 
prepared  for  Panel  1  use,  show  the  duplication  of  effort,  the  lack  of 
clarity,  etc.  It  is  quite  clear  that  a  complete  listing  of  the  various 
DoD  publications  of  QARAM  would  assist  in  reducing  the  size  of  the 
communication  problem. 

It  is  apparent  that  the  quality  and  utility  of  a  QARAM  Handbook  will 
depend  to  a  large  extent  on  the  capability  of  the  people  assigned  to  its 
preparation.  High  level  QARAM  talent  must  be  made  available. 


Recommendation 

It  is  recommended  that  the  OSD  authorize  and  direct  the  preparation 
of  an  administrative  handbook  covering  QARAM  and  other  functions 
associated  with  these  disciplines.  The  handbook  should  include  policy, 
general  procedures,  management  concepts,  and  techniques,  a  complete 
glossary  of  terms  for  mandatory  use  by  the  services,  and  an  appendix 
listing  all  of  the  directives  and  implementing  instructions  in  th  '  QARAM 
field.  Further  that  the  original  preparation  and  the  subsequent  up-dating 
necessary  for  continued  usefulness,  be  accomplished  by  personnel  with 
the  requisite  level  of  experience  and  technical  capability.  Further,  that 
a  target  date  of  1  July  1967  be  established  for  first  publication  of  the 
handbook. 
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New  Concepts  on  Emphasis 

The  interface  between  maintainability  and  maintenance  engineer¬ 
ing  has  not  been  made  clear  by  the  DoD.  Maintainability  is  the  restora- 
bility  of  product  attributes  within  the  specified  expenditure  of  resources 
(time,  money,  manpower,  facilities  or  materials)  expressed  as  a  prob¬ 
ability.  Maintenance  engineering  is  a  discipline  that  provides  effec¬ 
tive  and  economical  maintenance  support  through  the  optimum  use  of 
available  resources.  It  is  conceivable  that  the  optimum  maintainability 
design  would  not  be  optimum  design  from  a  maintenance  engineering 
point  of  view. 

When  these  goals  are  approached  by  separate  functional  elements 
with  no  clear  cut  definition  of  responsibility,  designers  are  often  faced 
with  conflicting  suggestions,  both  intended  to  enhance  the  ability  to 
maintain.  Through  evolution,  the  newer  discipline  of  maintainability 
has  been  permitted  to  acquire  a  meaning  more  descriptive  of  repair - 
ability . 


Recommendation 

It  is  recommended  that  OSD  recognize  the  apparent  difficulties 
associated  with  implementing  a  discipline  that  has  been  allowed  to 
transgress  into  both  the  design  function  and  the  support  function  by 
clearly  communicating  in  applicable  documents  its  definitions  of 
functions  that  are  clearly  within  the  domain  of  maintainability  and 
those  that  belong  to  maintenance  engineering. 

6,  Information  Gathering  and  Dissemination 

There  is  a  need  for  better  access  to  information  about  QARAM 
capability  at  various  Government  activities.  An  effective  system  should 
be  worked  out.  It  would  provide  a  communication  mode  of  great 
importance--particularly  between  DCASR's  and  using  activities.  The 
Personnel  Automatic  Data  Systems  (PADS)  service  (Navy)  contains 
some  information  on  personnel  qualification.  Perhaps  by  building 
PADS,  a  satisfactory  solution  could  be  obtained.  The  Central  Inven¬ 
tory  Management  System  (CIMS)  effort,  while  useful  in  the  long  term 
is  too  comprehensive  in  nature  to  be  available  within  a  reasonable  time 
and  is  not  really  oriented  towards  skill-training  information  acquisition. 
The  Defense  Documentation  Center  (DDC)  effort  is  not  effective  yet. 

There  is  a  clearly  established  need  to  acquire,  retrieve,  and 
retain  information  concerning  vendor  pre-award  surveys  by  various 
Governmental  activities. 
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Many  such  pre-award  surveys  are  now  being  made.  Often  the 
survey  teams  cross  each  other's  path  within  the  vendor's  plants  or  his 
offices.  It  appears  very  necessary  that  a  common  system  be  estab¬ 
lished  to  reduce  the  Government's  expenditures  in  this  area,  and  to 
reduce  the  problems  of  the  contractor's  in  having  so  many  Govern¬ 
mental  activities  reviewing  their  operations  in  the  same  area. 

There  is  a  need  for  a  vendor  performance  rating  and  retrieval 
system.  Vendors  have  often  been  surveyed  and  received  acceptable 
marks.  Their  performance,  however,  is  often  unsatisfactory.  It 
appears  that  it  is  very  desirable  to  establish  a  system  of  vendor  per¬ 
formance  ratings  and  to  provide  a  means  to  retrieve,  retain  and  use 
such  information. 

The  CPE  (Contractor  Performance  Evaluation)  program  that  has 
recently  been  initiated,  is  a  good  beginning.  CPE  could  be  further 
developed  to  include  a  rating  system  which  would  provide  through  a 
relatively  simple  combination  of  digits,  a  rating  based  upon  the  ven¬ 
dors  performance  in  producing  products  for  the  DoD. 

There  is  a  requirement  for  a  vendor  capability  retrieval  system. 
All  vendors  are  not  alike.  Vendors  do  not  retain  the  same  capability 
year  in  and  year  out  as  the  management  changes  the  direction  of  the 
company  and  as  technology  produces  its  impact.  There  is  a  need, 
therefore,  to  develop  information  and  retrieve  and  store  such  infor¬ 
mation  concerning  vendor  capabilities. 

There  are  many  thousand  of  vendors  of  commodities,  equipments, 
systems  and  subsystems  delivering  products  to  the  DoD.  These  ven¬ 
dors  of  all  sizes  and  descriptions  provide  products  to  one  or  more 
services  at  different  times.  Information  resulting  from  surveys  and 
audits  is  sometimes  available  to  other  activities,  but  seldom  is  avail¬ 
able  in  sufficient  detail  and  sufficiently  categorized  so  as  to  be  useful 
to  other  DoD  activities.  Therefore,  it  would  appear  that  significant 
advantage  could  be  obtained  for  the  DoD  and  substantial  cost  reduction 
obtained  if  a  common  vendor  capability  rating  system  could  be  devel¬ 
oped  and  the  information  provided  therein  transmitted  to  a  central 
recording  activity.  Such  central  recording  activity  would  be  a  memory 
bank  which  could  be  queried  by  DoD  and  other  Governmental  activities 
to  determine  what  rating  a  particular  vendor  has. 

Obviously,  such  a  central  bank  of  information  would  have  to  have 
the  capability  of  being  continuously  updated,  continuously  monitored, 
and  continuously  checked  for  uniformity.  In  addition,  for  highly  com¬ 
plex  systems  and  high  consequence  of  failure  types,  the  system  snould 
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provide  information  as  to  how  a  vendor  actually  performs  in  relation 
to  specific  contracts. 


Recommendations 


It  is  recommended: 

1.  That  OSD  establish  and  maintain  a  responsive  system  so  that 
information  about  the  QARAM  capability  of  Gover.  ment  activities  is 
collected,  stored,  and  disseminated  as  needed  for  efficient  and  eco¬ 
nomical  utilization  of  these  important  resources, 

2.  That  the  OSD  develop  a  system  for  making  pre -award  survey 
information  readily  available  to  all  Governmental  activities  in  a  timely, 
useful  manner. 

3.  That  the  OSD  study  the  desirability  of  expanding  the  Contractor 
Performance  Evaluation  (CPE)  Program  to  include  the  development  and 
use  of  a  common  vendor  performance  rating  system  based  on  numbers 
rather  than  adjectives. 

4.  That  the  OSD  study  the  matter  of  a  common  vendor  capability 
rating  system  and  having  determined  the  necessary  elements  of  such 

a  rating  system  provide  for  the  acquisition  of  such  information  in  a 
central  receiving  center,  such  receiving  center  to  have  memory  capa¬ 
bility,  inquiry  capability  and  transmission  to  inquirer  capability.  To 
the  maximum  extent  possible,  the  system  should  contain  information 
about  how  the  vendor  actually  performed,  not  just  his  apparent  capa¬ 
bility  of  performance. 

7.  Standard  Product  History  Information 

In  the  diverse  DoD  product  area  where  products  are  being 
produced  by  many  thousands  of  vendors  for  many  different  organi¬ 
zations  within  the  armed  services,  with  many  of  the  same  basic  build¬ 
ing  blocks  such  as  transistors,  O-rings,  copper,  nickel,  etc.  ,  there 
exists  the  need  for  a  central  bank  of  QAR.AM  information  which  would 
pool  the  various  fragmented  elements  currently  available  to  the  Gov¬ 
ernmental  activity  product  users.  Such  a  bank  of  information  would 
be  of  major  importance  if  it  could  be  provided  in  such  a  way  as  to 
insure  prompt,  up-to-date  transmission  cf  answers  to  queries  about 
the  characteristics  of  product  as  acquired,  as  stored,  as  used. 

It  is  fully  recognized  that  the  development  of  such  a  bank  of  infor¬ 
mation  would  be  extremely  difficult  and  after  experience  with  the 
Federal  stock  numbering  system,  it  might  prove  to  be  too  difficult  to 
provide  a  full  range  of  information  across  the  entire  spectrum  of  prod¬ 
ucts  used  by  the  DoD  and  the  vendors  involved  in  supplying  them.  It 
would  be  possible,  however,  for  a  rcntral  bank  of  information  to  be 
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established  by  technical  disciplines  which  could  provide  information 
on  vitally  important  areas  and  on  areas  of  major  cost  applications.  If 
the  information  bank  could  be  restricted  in  its  initial  application  and  if 
it  could  provide  prompt  response,  major  savings  could  be  accrued. 

It  must  be  pointed  out,  however,  that  unless  the  service  is  prompt 
and  effective,  that  it  will  be  useless  and  could  be  a  detriment  to  the 
operation. 

There  is  a  need  within  the  Government  agencies  of  DoD  for  a 
uniform  product  deficiency  retrieval  system.  When  a  product  defi¬ 
ciency  of  some  significance  occurs,  an  abnormal  amount  of  attention 
is  frequently  focused  on  what  may  be  an  isolated  incident.  By  utili¬ 
zing  computer  capability,  we  could  exert  management  control  to  pre¬ 
clude  recurrence  and  ascertain  actual  performance  levels  of  in-use 
products. 

Product  deficiency  retrieval  systems  exist  in  some  form  in  most 
Government  agencies,  but  top  management  attention  has  not  beer, 
focused  on  the  problem  of  disseminating  product  deficiency  data  to 
concerned  DoD  entities. 


Recommendations 


It  is  recommended: 

1.  That  OSD  study  the  advisability  of  instituting  a  data  collection 
cperation  in  the  area  of  vendor  QARAM,  such  data  collection  to  be 
housed  in  a  central  information  bank,  available  to  using  activities, 
with  particular  emphasis  on  vendor  QARAM  techniques. 

2.  That  OSD  study  the  desirability  of  establishing  a  uniform  prod¬ 
uct  deficiency  retrieval  system. 

8.  Quantitative  Expressions 

One  of  the  basic  difficulties  in  communicating  DoD  and  head¬ 
quarters  QARAM  policy  and  directives  is  that  words  have  entirely  dif¬ 
ferent  connotations  in  the  various  levels  of  society  within  the  United 
States,  within  the  various  technical  disciplines  within  the  United  States, 
and  indeed  within  the  various  Governmental  circles  within  the  United 
States.  As  a  result  the  directives,  policy  statements  and  implementing 
instructions  put  out  by  the  DoD  and  its  subordinate  activities  require 
more  quantitative  expression  to  be  sufficiently  clear  to  insure  reason¬ 
able  understanding  and  adherence. 
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There  are  a  number  of  ways  to  solve  this  problem,  but  the  most 
likely  provides  for  the  use  of  mathematical  language  wherever  such 
language  can  be  effectively  used. 

The  arithmetical  numbers  -  1,  2,  3,  4,  ....  mean  the  same  to 
all  of  us.  It  is  possible  through  the  use  of  numbers  to  convey  ideas 
which  are  far  more  precise  than  descriptions.  Further,  equations, 
whether  they  be  algebraic  or  otherwise,  can  provide  a  full  and  com¬ 
plete  understanding  from  one  person  to  another.  Much  of  the  prob¬ 
lem  of  QARAM  is  linear  in  nature,  in  fact  is  relatively  simple  in 
nature  in  that  one  or  more  elements  can  be  adequately  described  with 
an  arithmetical  series  of  numbers.  Other  elements,  however,  such 
as  in  highly  complex  systems,  may  require  integral  or  differential 
equations,  or  application  of  higher  mathematical  techniques. 

Most  of  the  chemical,  biological,  physical  phenomena  which  are 
used  in  products  are  described  by  equations;  most  of  man's  application 
of  energy  in  machines  to  products  can  be  described  by  equations;  for 
example,  computer  type  lathes  are  tape-fed  machines,  clearly  gov¬ 
erned  by  equations.  Thus,  it  appears  quite  possible  that  many  of  our 
QARAM  type  functions  could  be  described  by  a  mathematical  type 
language.  By  provision  of  such  mathematical  language,  the  inaccu¬ 
racies  would  be  sharply  reduced;  the  understanding  sharply  increased. 
There  are  many  possibilities  in  this  regard. 


Recommendation 

It  is  recommended  that  OSD  encourage  the  use  of  mathematical 
expressions  wherever  practicable  in  the  communication  chain  of 
QARAM. 

9.  Communication  Means 

We  have  recognized  the  many  difficulties  in  communication 
that  arise  from  the  various  definitions  and  connotations  found  in  the 
written  English  word.  It  is  apparent  that  even  with  our  best  efforts 
to  improve  and  clarify  written  definitions  and  procedures,  we  will 
still  have  problems  in  communication.  One  of  the  most  effective 
methods  of  insuring  complete  understanding  and  reasonable  uniformity 
in  carrying  out  DoD  directives  is  the  face  to  face  discussion  provided 
by  conferences.  Management  must  provide  guidance  and  direction 
through  written  documentation,  but  management  also  must  encourage 
optimum  use  of  verbal  communication  to  insure  that  uniform  inter¬ 
pretation  is  being  made. 
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Recommendation 


It  is  recommended  that  OSD  make  optimum  use  of  face  to  face 
discussions  and  conferences  to  insure  that  QARAM  management  at  all 
levels  has  communicated  effectively. 


JO 


ORGANIZATION 


1.  Organizational  Concepts 

One  of  the  major  difficulties  in  creating,  developing,  pro¬ 
ducing,  storing,  maintaining,  and  using  diverse  products  is  how  to 
organize  the  different  elements  in  this  total  product  life  cycle.  Obvi¬ 
ously,  the  same  organization  that  would  be  useful  for  thinking  would 
not  necessarily  be  useful  for  using,  nor  for  storing,  and  vice  versa. 

The  strong  elements  of  disciplined  thinking  which  are  so  vital  in  the 
early  stages  of  a  product's  life  cycle  become  less  important  while 
the  aspects  of  disciplined  doing  become  vitai  in  the  storage  and  use 
portion  of  the  life  cycle.  Different  organizations  are  needed  and  dif¬ 
ferent  distributions  of  resources  are  required. 

In  looking  to  organizational  patterns,  one  finds  no  universal 
pattern  which  would  satisfy  these  diverse  needs.  It  therefore  appears 
that  the  organizations  and  organizational  setups  which  are  determined 
to  be  optimum  for  the  various  parts  of  the  life  cycle  must  contain  iden¬ 
tification  and  auditability  for  the  important  technical  disciplines  that 
are  involved.  QARAM  disciplines  should  be  identified  through  DoD 
organizations  from  the  commander  to  the  final  user  of  the  product. 
Identification  in  this  case  means  that  important  elements  of  QARAM 
are  written  into  the  organization  charts  to  show  a  continuous,  auditable 
chain  of  responsibility  from  the  top  cf  the  organization  to  the  user  of 
the  end  product.  The  QARAM  function  in  most  activities  will  be  that 
of  monitoring  and  enforcement;  even  in  engineering  activities  such  a 
function  is  present  in  addition  to  the  QARAM  disciplines  contouring 
effect  on  product  attributes. 

In  recent  years  there  has  been  an  increasing  tendency  for  splinter 
disciplines  to  break  away  from  the  basic  engineering  of  products.  For 
example,  value  engineering,  safety  engineering,  human  engineering, 
and  the  like  have  been  removed  from  the  engineering  organization  of 
many  vendors,  many  Government  activities,  and  set  up  in  separate 
small  entities  outside  of  the  coordinating  influence  of  the  supervisors 
of  the  engineering  function.  While  there  can  oe  no  question  that  empha¬ 
sis  to  these  important  engineering  disciplines  must  be  applied,  through 
spectacular  techniques  from  time  to  time,  the  continuation  of  the  pro¬ 
liferation  of  such  splinter  disciplines  into  new  organizational  units  can 
only  result  in  degradation  of  the  final  product  over  the  long  term. 
Indeed,  today,  we  see  evidence  that  these  separate  units,  in  a  fight 
for  survival  past  their  time  of  usefulness,  currently  are  aggrandizing 
unto  themselves  other  engineering  functions  which  are  not  concerned 
with  their  splinter  discipline.  Today,  for  example,  the  value 
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engineering  function  is  aggrandizing  management  engineering  areas. 

The  Zero  Defects  concept  splinter  area  is  aggrandizing  various  areas 
of  technical  responsibility.  The  splinter  disciplines  seldom  provide 
balanced  reactions  to  problems,  but,  rather,  proposed  solutions  which 
highlight  their  particular  discipline  at  the  expense  of  the  over-all 
effectiveness  and  efficiency  of  the  product  desired. 

There  is  a  vital  need  that  the  engineering  function  be  considered  in 
total.  All  of  the  basic  disciplines  must  be  fully  considered  for  each 
product.  No  discipline  should  be  overemphasized  in  the  product,  merely 
because  a  strong  organizational  element  has  been  able  to  subvert  the 
basic  product  purpose  with  emphasis  on  the  splinter  discipline.  Inte¬ 
grated  engineering  is  a  vital  necessity  for  effective  and  efficient  prod-  ^ 
ucts. 

It  is  particularly  important  that  the  integrated  engineering  concept 
be  considered  in  the  organization  of  field  activities,  that  is,  producers, 
overseers,  and  users.  It  appears  that  the  OSD  and  the  military  depart¬ 
ment  headquarters  in  Washington  could  profit  by  review  of  this  inte¬ 
grated  engineering  principle  v/ith  the  expectation  that  many  of  the 
splinter  disciplines  which  have  been  prolonged  too  long,  oversold  by 
brochuremausHp,  and  press  agentry,  could  be  put  back  into  the  over¬ 
all  engineering  area  where  they  belong. 

There  can  be  no  question  that,  from  time  to  time,  certain  contour¬ 
ing  engineering  discipline  elements  require  additional  emphasis.  When 
such  emphasis  is  supplied,  however,  through  the  creation  of  separate 
units,  separate  projects,  etc.  ,  it  is  mandatory  that  the  provision  fer 
the  reamalgamation  of  that  discipline  and  its  associated  people  back 
into  the  integrated  engineering  group  or  organization  be  provided. 


Recommendations 


It  is  recommended: 

1.  That  OSD  require  organizational  identification  and  auditability 
for  QARAM  disciplines  in  the  field  activities. 

2.  That  OSD  include  in  its  organizational  concepts  a  basic 
requirement  that  integrated  engineering  concepts  be  adopted  by  field 
activities  and  that  the  organization  of  field  activities  adequately  con¬ 
form  and  implement  this  policy;  that  the  OSD  provide,  in  establishing 
projects  or  units  to  emphasize  splinter  engineering  disciplines,  the 
time  and  the  mode  for  full  integration  of  these  splinter  disciplines  into 
the  basic  engineering  conceptual  and  organizational  operations. 
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2.  Washington  Headquarters  Organizations 


The  current  OSD  organization,  particularly  its  DDR  and  E 
and  Installations  and  Logistics  components,  appears  satisfactory  to 
support  the  QARAM  effort  if  interface  conditions  are  established  and 
supported. 

Product  attributes  are  the  result  of  application  of  basic  engineer¬ 
ing  disciplines  and  contouring  disciplines  to  the  product  concept. 

Since  determination  of  product  attributes  is  an  engineering  function, 
and  quality  is  "a  condition  achieved  by  meeting  specified  product  attri¬ 
bute  requirements, 11  engineering  establishes  product  quality  require¬ 
ments.  Thus,  the  OSD  (DDR&E)  is  responsible  for  setting  product 
QARAM  requirements,  including  software. 

Attainment  of  QARAM  requirements  through  control  and  assurance 
disciplines  is  the  responsibility  of  OSD  (I&L). 

The  OSD  headquarters  organization  is  composed  of  a  series  of 
separate  individual  units  with  no  formal  intercommunication  or  inter¬ 
cognizance  assignments.  Such  cooperation  and  interplay  as  may  be  the 
result  of  knowledgeable  personnel  or  friendships  do  transpire  within 
any  large  organization.  There  is,  however,  the  need  from  the  DoD 
for  a  formalized  interplay  and  for  coordination  to  develop  the  best 
over-all  direction,  rather  than  overemphasis  on  one  of  the  minor 
elements.  Within  the  headquaiters  organization  of  the  OSD.  there  is 
a  clear  need  for  an  audit  capability  to  internally  audit  the  OSD.  Such 
internal  audits  would  have  as  a  prime  purpose,  determination  of  work¬ 
ing  arrangements  as  have  already  been  definitized.  Such  audit  would 
also  disclose  the  many  areas  in  which  coordination  is  lacking  and  in 
which  the  directives  which  are  forwarded  to  field  activities  are  clearly 
not  optimized  for  the  good  cf  the  DoD,  but  rather  for  the  good  of  the 
OSD  activity  which  generated  the  particular  directive.  The  audit  capa¬ 
bility  in  the  OSD  (Comp)  is  not  considered  adequate  for  this  function. 

The  OSD  needs,  in  addition  to  its  capability  to  audit  itself  inter¬ 
nally  at  the  headquarters,  the  capability  of  auditing  the  field.  Without 
such  audits,  there  is  no  way  that  the  managemeni  can  really  determine 
whether  its  directives  are  in  fact  being  carried  out,  whether  in  fact 
they  are  being  understood,  whether  in  fact  they  can  be  carried  out. 

The  various  military  departments  do  provide  some  audit  elements,  for 
example,  from  the  Navy  and  other  military  audit  agencies,  from  the 
GAO,  ?.b  well  as  frem  special  on-the-spot,  often  quick  reviews  of  small 
portions  of  problems.  There  is,  however,  no  complete  audit  of  sys¬ 
tems,  /'rganizations  or  disciplines.  While  it  is  true  that  such  an  audit 
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plan  could,  if  improperly  implemented,  result  in  a  considerable  amount 
of  additional  work  in  the  agencies  of  the  DoD,  it  is  also  true  that  a  well 
organized  plan  could  reduce  the  ;otal  amount  of  effort  required  by  mil¬ 
itary  departments  to  support  the  OSD's  efforts.  It  appears,  therefore, 
that  there  could  be  substantial  improvement  in  QARAM  operations,  at 
lower  cost,  if  the  OSD  had  the  capability  for  audit  of  its  own  activities. 

Military  department  organizations  au  the  Washington  level  have 
been  made  more  uniform  during  the  past  few  years  in  their  titles  and 
their  functions.  The  comments  relating  to  the  OSD  QARAM  organ¬ 
izational  structure  should  apply. 

Functional  and  organizational  identification  and  auditability  of  the 
QARAM  function  is  required. 


Recommendations 


It.  is  recommended: 

1.  That  OSD  reaffirm  that  determination  of  product  QARAM  is 
the  function  of  OSD  (DDR&E).  The  fulfillment  of  the  QARAM  require¬ 
ments  is  the  function  of  OSD  (I&L). 

2.  That  OSD  initiate  a  study  to  determine  the  desirability  and 
organizational  position  of  an  auditing  (management  type)  group  which 
would  internally  audit  the  OSD  headquarters  operations. 

3.  That  DoD  develop  and  establish  the  capability  for  auditing  the 
field  activities  in  the  QARAM  areas. 

4.  That  the  military  Service  headquarters  in  Washington  adopt 
organizational  structures  which  provide  full  identification  and  audit- 
ability  for  the  QARAM  function. 

3.  Field  Headquarters  Organizations 

Field  headquarters  organizations  are  generally  subordinate 
elements  based  on  logistics,  product  support,  etc.  As  a  result,  they 
are  generally  organized  to  provide  support  of  the  basic  mission.  They 
generally  have  and  require  all  of  the  elements  necessary  for  directing, 
overseeing,  and  managing,  and  occasionally  producing  various  products 
and  systems.  As  such,  they  require  all  of  the  engineering,  production, 
resource  management,  and  staff  discipline  capabilities  as  are  required 
within  such  places  as  the  OSD  headquarters,  military  system  head¬ 
quarters,  etc.  An  important  resource  which  is  required  in  field  head¬ 
quarters  organization  is  that  relating  to  support  of  the  QARAM  disci¬ 
pline.  There  is  a  clearly  indicated  need  for  a  top  level,  highly  skilled 
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staff  whose  function  is  QARAM;  to  be  available  to  the  field  headquarters 
commander  to  provide  advice  and  assistance  in  the  attainment  of  QARAM 
objectives.  The  identification  and  auditability  of  the  QARAM  discipline 
should  be  clear  in  subordinate  organizations. 


Recommendation 

It  is  recommended  that  OSD  review  the  various  kinds  of  field  head¬ 
quarters  organizations  to  determine  mission  and  insure  that  the  basic 
element  and  discipline  of  QARAM  is  provided  at  the  commander's  level 
in  these  headquarters  with  full  identification  and  auditability  in  the 
subordinate  organizations. 

4.  Field  Component  Organizations 

Field  activities  are  usually  involved  with  a  specific  phase  of 
the  life  cycle  of  a  product,  such  as  development  or  storage,  or  use. 
These  units  should  generally  be  organized  on  a  phase-mission  basis, 
rather  than  on  a  technical  or  scientific  discipline-type  basis,  but  with 
staff  organization  having  discipline  elements  in  it  which  support  and 
advise  the  basic  project  leadership.  What  this  means  is  that  an  orga¬ 
nization  in  the  field  should  generally  have  engineering,  production, 
resource  management,  and  various  supporting  staff  organizations,  all 
organized  to  most  effectively  and  efficiently  use  the  products  which  are 
the  basis  for  the  activity's  very  life.  The  disciplines  of  QARAM  should 
be  organized  as  identifiable  and  auditable  elements  through  the  organi¬ 
zation. 


Recommendation 

It  is  recommended  that  field  activities  be  organized  along  product 
lifephase  lines  to  support  the  major  mission,  rather  than  be  organized 
by  scientific  or  technical  discipline,  and  that  QARAM  be  an  identifiable 
and  auditable  element  therein. 

5.  Vendors'  Organizations 

Vendors  of  products  to  the  DoD  are  generally  private  concerns. 
They  generally  organize  themselves  for  maximum  effectiveness  and 
efficiency  around  a  product  or  a  particular  discipline  which  they  vend 
as  a  service.  Their  QARAM  organizations  are,  therefore,  almost 
always  dictated  by  the  kind  of  product  or  service  they  sell.  The  DoD, 
however,  in  acquiring  such  products,  often  uses  them  as  elements  of 
a  subassembly  or  as  component  elements  of  a  larger  system  assembly. 
The  DoD  QARAM  requirements  are  often  written  to  the  larger  product, 
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the  system,  rather  than  to  the  elements  of  the  components  and  the  ven¬ 
dors  thereby  are  often  required  to  produce  a  QARAM  organization  or 
system  which  is  not  in  consonance  with  the  requirements  of  their  prod¬ 
ucts. 

It  appears,  therefore,  that  the  DoD  should  avoid,  where  possible, 
specific  definition  of  the  organization  or  the  methodology  to  be  used  by 
a  specific  vendor  of  components  and  small  portions  of  systems,  but 
instead  should  require  that  the  scope  of  QARAM  be  consistent  with 
products'  intended  use. 


Recommendation 

It  is  recommended  that  OSD,  as  a  part  of  the  QARAM  audit  program 
assure  the  proper  level  of  QARAM  application  in  vendor's  organization. 


RESOURCES 


1.  Resource  Allocation 

Many  vendors  who  supply  equipments  and  components  to  a 
multiplicity  of  DoD  contractors  are  often  being  monitored  by  several 
different  contractors  at  the  same  time.  This  surveillance  is  time- 
consuming,  confusing,  and  results  in  unnecessary  duplicative  effort. 

As  a  result,  many  vendors  have  suggested  that  industry  come  up  with 
a  plan  for  conducting  audits  of  plants  by  unbiased,  separately  organized 
activities,  that  these  audits  then  be  taken  as  the  official  determination 
of  the  capabilities  of  the  vendor  and  that  these  audit  findings  be  recog¬ 
nized  by  the  DoD.  These  so-called  United  Vendors  Organizations,  such 
as  AIA,  NSIA,  CODSEA,  etc. ,  would  then  be  in  a  position  of  policing 
themselves,  somewhat  similar  to  a  Better  Business  Bureau.  It  is 
believed  that  this  concept  is  sound  and  it  would,  perhaps,  provide  the 
basis  for  far  less  industry  monitoring  of  vendors,  resulting  in  perhaps 
an  occasional  audit  rather  than  continuing  surveillance.  Further,  it 
appears  that  as  a  result  of  such  audits  and  acceptance  by  an  entire 
industry  of  similar  standards  that  the  total  use  of  QARAM  personnel 
could  bt  more  effectively  directed  within  industry  and  Government. 

This  kind  of  approach  has  the  greatest  attraction  in  areas  which  are 
ordinarily  commercial  in  nature  or  have  a  large  application  to  many 
products  with  DoD  needs.  It  is  noted  that  DoD  already  often  uses  in 
its  specifications  the  rules  of  various  private  technical  groups,  such 
as  ASTM,  ASME,  etc.  A  combined  DoD-Industry  study  would  reveal 
areas  of  possible  resource  saving  through  this  approach. 

The  allocation  of  QARAM  resources  is  spotty,  and  dependent  largely 
upon  the  understanding  by  the  individual  commands  of  the  necessity  for 
the  QARAM  functions.  Thus,  it  apprars  that  a  more  effective  utilization 
of  these  resources  could  be  made  with  the  ultimate  improvement  in 
product  or  alternatively,  reduction  in  costs  at  the  same  QARAM  level. 

A  large  portion  of  the  QARAM  personnel  making  up  these  resources 
are  near  to  retirement  age  due  to  the  World  War  II  and  Korean  War 
"hump.  "  A  very  serious  crippling  of  the  Government's  capability  could 
occur  if  plans  are  not  made  in  advance  to  acquire  or  relocate  other 
personnel. 

Acquisition  of  information  about  such  resources  is  a  difficult  task, 
but  one  which  would  not  prove  impossible  if  attacked  with  vigor.  It 
appears  that  systematic  acquisition  of  information  about  QARAM 
resources  (mainly  people)  should  be  initiated  to  provide  a  knowledge 


37 


bank  of  these  resources  and  their  capabilities.  It  is  understood  that  the 
Defense  Documentation  Center  intends  to  provide  information  in  this 
area,  but  at  a  later  date.  Such  information  is  needed  now. 

Following  acquisition  of  the  knowledge  bank,  allocation  of  the 
resources  could  be  better  made,  provided  other  information  as  to  the 
need  for  such  resource  reallocation  has  been  made  known  through  the 
central  information  gathering  and  dissemination  facility  discussed  under 
C  ommuni  c  a  ti  on . 


Recommendations 


It  is  recommended: 

1.  That  OSD  request  industrial  groups,  such  as  AIA,  NSIA,  etc.  , 
to  join  in  a  study  to  determine  the  usefulness  of  vendors'  self -policing 
in  the  QARAM  field  and  recommend  ways  to  reduce  duplication. 

2.  That  OSD  study  the  QARAM  resources  available  with  the  under¬ 
standing  that  what  is  needed  first  is  knowledge  and  second,  possible 
reallocation  to  acquire  better  utilization  of  these  resources. 

2.  Resource  Retention 

Personnel  in  the  QARAM  area  do  not  have  career  patterns 
which  lead  to  maximum  incentive  nor  maximum  flexibility  for  effective 
use  in  the  various  organizations  within  the  DoD.  Because  of  this,  the 
personnel  do  not  find  ways  to  move  from  one  organization  to  another; 
their  capabilities  are  not  always  well  utilized,  but  most  importantly, 
all  too  often  they  seek  advancement  or  challenge  by  going  outside  of  the 
organization.  It  appears  then  an  overall  listing  of  QARAM  trained  people 
would  help  to  a  large  extent  in  making  known  these  resources  within 
the  DoD,  but  equally  important,  it  could  form  the  basis  for  the  beginn¬ 
ing  of  a  rotational  plan  for  these  personnel;  such  a  rotational  plan  would 
involve  movement  from  one  activity  to  another  as  necessary  to  fully 
utilize  their  skills  and  to  make  possible  for  them  an  advancement  ladder 
which  would  be  attractive  in  terms  of  different  types  of  projects,  con¬ 
cerns,  different  geographical  locations,  different  aspects  of  QARAM. 
Personnel  retention  requires  heavy  use  of  incentive.  Very  few,  if  any, 
incentives  are  visible  today. 

The  loss,  in  a  few  years,  of  large  numbers  of  Government  QARAM 
personnel  adds  urgency  to  this  problem. 


3H 


Recommendations 


It  is  recommended  that  OSD  study  the  problem  of  personnel  retention 
in  the  QARAM  area  and  develop  a  series  of  incentives,  including  career 
patterns. 

3.  Resources  Acquisition 

There  is  a  need  to  acquire,  train  and  maintain  highly  skilled, 
highly  motivated  personnel  with  a  QARAM  background.  These  personnel, 
who  do  not  exist  in  sufficiently  large  numbers  in  the  United  States,  are 
mainly  concentrated  in  industrial  areas  involving  mass  production, 
multiple  production  and  only  to  a  limited  extent  in  industries  which  have 
low  single  unit  production  runs.  Many  of  the  DoD  products  are  one  of 
a  kind  or  have  extremely  few  numbers  in  prodution  runs,  are  often 
modified  extensively  as  production  is  taking  place  and  many  times  are 
highly  complex  Under  such  circumstances  it  is  vitally  necessary  that 
the  QARAM  personnel  be  of  superior  calibre. 

The  OSD  establishment  in  Washington  needs  additional  professional 
QARAM  resources  in  its  activities  and  operations.  In  addition,  other 
activities  require  additional  QARAM  professionals.  These  personnel 
are  in  short  supply  in  industry  as  well  as  in  the  DoD  field  activities. 

It  appears,  therefore,  that  a  training  program  will  be  required  to  aug¬ 
ment  the  numbers  to  the  proper  number  required. 


Recommendation 

It  is  recommended  that  DoD  increase  its  efforts  to  obtain  and  retain 
high  quality,  high  performance  personnel  qualified  in  QARAM. 
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DISCIPLINE 


The  fifth  definition  of  discipline  in  the  American  College  Dictionary 
is  as  follows:  "a  state  of  order  maintained  by  training  and  control. " 
These  simple  words  define  a  condition  which  presupposes  a  number  of 
acts  have  taken  place. 

The  maintenance  of  a  state  of  order  indicates  that  there  was  a 
concept,  a  development  of  a  plan,  a  mode  for  implementation,  a  promul¬ 
gation  of  a  plan,  an  execution  of  a  plan,  an  auditing  of  a  plan,  and  the 
improvement  of  a  plan;  that  these  acts  were  performed  by  personnel 
of  requisite  skills,  background,  training,  and  desire.  Thus,  discipline, 
as  used  in  this  context,  involves  the  total  operation  of  doing  something 
and  doing  it  well. 

It  is  an  element  which  is  vital  to  the  success  of  any  human  endeavor. 
Some  endeavors  require  more  effort  than  others,  some  can  be  done  with 
little  or  no  planning,  with  little  or  no  need  for  teamwork  in  execution, 
with  little  or  no  need  for  review  of  the  act.  But  the  areas  in  which  the 
DoD  is  concerned  are  increasingly  complex,  high  cost,  involving  many 
humdreds  if  not  thousands  of  people  in  a  very  complicated,  difficult 
interplay  of  individual  effort,  interaction  of  machines  and  facilities 
under  increasingly  shorter  time  restraint.  It  is  in  this  kind  of  atmo¬ 
sphere  that  the  DoD  must  consider  the  "discipline"  discipline. 

QARAM  has  no  good  plan  because  of  the  immense  complexity  of  the 
DoD  spectrum  of  products,  life  cycle  of  products,  personnel  available, 
facilities  and  equipments  available.  No  plan  yet  exists  because  of  the 
problems  of  working  into  one  plan  all  of  these  elements. 

No  plan  can  exist  or  be  developed  until  there  is  a  basic  understand¬ 
ing  that  there  must  be  the  use  of  general  principles  of  man's  behavior 
rather  than  specific  application  of  one  or  more  of  the  minor  techniques 
or  engineering  disciplines.  Nonetheless,  an  effort  must  be  made  as 
early  as  possible  to  provide  a  general  plan  and  to  provide  elaborations 
of  that  plan  for  areas  of  vital  importance  at  the  present  time. 

It  appears  that  the  DoD  should  begin  its  considerations  of  QARAM 
as  a  major  contouring  discipline  which  biases  the  attributes  of  the 
products  which  are  desired.  QARAM  includes  disciplines  which  involve 
the  entire  system  of  which  the  product  is  a  part  as  well  as  the  product 
itself.  They  relate  to  maximum  system  effectiveness  and  minimum 
system  cost  rather  than  maximum  product  effectiveness  and  minimum 
product  cost.  The  plan  which  is  developed  for  QARAM  must  take  into 
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account  all  of  the  elements  of  the  life  cycle;  and  must  equally  well  take 
into  account  the  probabilities  of  the  capabilities  of  the;  people  to  use-  the 
product  and  the  system  of  which  it  is  a  part. 

Only  bits  and  pieces  of  such  a  plan  relating  to  some  elements  of  the 
overall  problem  exist.  These,  as  one  would  well  imagine,  arc*  partial 
solutions  and  often  involve  overemphasis  of  certain  elements  of  the 
problem.  To  have  discipline,  a  complete  plan  must  be  developed. 

Immediately  after  the  concept  has  been  developed  and  a  plan  organ¬ 
ized  for  putting  it  into  effect,  there  is  need  for  development  of  the 
implementation  phase.  This  development  requires  that  the  entire  con¬ 
cept  of  the  plan  be  tested  by  mental  or  actual  simulation  prior  to  being 
put  into  operation,  that  the  difficulties  of  putting  it  into  operation  be 
anticipated  and  ways  to  work  around  them  arranged,  that  the  individuals 
who  will  be  implementing  are  studied  to  determine  that  their  organic: 
tional  placement  and  personalties  are  such  that  implementation  will  be 
effective.  Implementation  must  be  phased  for  complex  products  so  that 
the  changeover  from  existing  to  new  can  take  place  with  minimum  orga¬ 
nizational  friction  and  confusion.  Indeed,  the  change  from  one  way  of 
doing  things  to  another  often  costs  corporations  and  Government  activi¬ 
ties  their  vitality  and  drive  and  capability- -in  some  cases,  never  to  be 
recovered,  and  in  other  cases  to  be  recovered  only  after  long  and  pain¬ 
ful  efforts  to  crawl  back  into  positions  of  prominence.  In  still  other 
cases  where  a  new  concept  is  invoked,  the  diversion  of  resources  during 
parallel  operations  from  the  old  organization  and  the  old  method  often 
seriously  cripples  the  activity. 

Implementation  is  a  vital  part  of  discipline.  It  is  a  part  which  is 
almost  consistently  left  out  in  the  OSD  directive  effort.  Seldom,  if 
ever,  is  there  an  indication  cf  trial  in  other  than  an  isolated  spot,  where 
the  plan  is  considered  to  have  the  best  possible  chance  of  success.  No 
overall  consideration  is  given  to  the  difficulties  of  implementing,  nor 
is  implementation  actually  thought  out.  This  serious  lack  is  fatal  in 
any  but  a  society  where  the  top  can  enforce  its  directives.  It  is  always 
detrimental  to  effectiveness  and  efficiency.  Implementation  of  OSD 
plans  needs  to  be  better  thought  out,  detailed,  rehearsed  and  made 
effective. 

Following  development  of  the  modes  of  implementation  of  a  plan, 
there  is  need  to  promulgate  it  in  a  way  which  is  clear  and  effective. 

This  usually  takes  the  form  of  directives,  handbooks,  specifications  and 
other  kinds  of  written  communications,  sometimes  implemented  by  face 
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to  face  meetings  and  conferences  to  discuss  and  outline  the  purpose, 
the  means  and  the  manner  in  which  the  plan  is  to  be  executed.  The 
promulgation  of  the  plan  is  an  important  element  for  all  too  often  the 
communications  from  the  development  of  the  plan  is  faulty,  leading 
to  poor  execution. 

The  execution  of  the  plan  usually  involves  field  activities  far 
removed  from  the  source  of  the  plan.  It  involves  thousands  of  individ¬ 
uals,  all  of  whom  have  a  different  approach  to  the  problem,  a  different 
understanding  of  the  communications  and  a  different  understanding  of 
the  plan  itself.  Thus,  the  execution  of  the  plan  may  or  may  not  be  in 
consonance  with  the  desired  goals  of  the  plan.  Execution  obviously 
involves  not  just  the  act  of  individuals  in  attempting  to  carry  out  the 
plan,  but  the  necessary  papers,  explanatory  memoranda,  and  other 
software  which  define  more  clearly  the  subdivisions  of  the  plan  into 
the  smallest  element.  Execution,  to  be  effective,  needs  to  be  primarily 
written  to  the  lowest  element  of  the  problem  and  plan  area;  the  actual 
do  becomes  a  smaller  and  smaller  portion  of  the  execution  as  the 
systems  become  more  complex.  The  think  becomes  more  important 
as  the  problem  becomes  more  complex.  The  actual  do  is  the  direct 
result  of  the  think. 

Since  the  execution  of  the  plan  may  not  be  in  conformance  with  the 
plans  or  ideas  of  those  who  generated  the  plans,  there  is  a  great  need 
for  audit,  such  audit  increasing  in  need  and  depth,  as  the  number  of 
activities  involved  increases,  as  the  complexity  of  the  product  increases 
and  as  the  complexity  of  the  environment  increases.  Audit  is  required 
to  determine  whether  the  plan  is  working,  is  in  fact  workable,  and  to 
give  the  director  of  the  plan  the  necessary  information  as  to  progress 
as  the  plan  is  executed.  This  feedback  of  information  often  provides  the 
basis  for  improving  the  plan  through  the  iterative  technique  causing 
development  of  changes,  i.  e.  ,  improvements,  promulgation,  execution 
and  audit  again.  Successive  re-cycling,  that  is,  successive  iterations, 
result  in  a  continuously  modified  plan  which  provides  maximum  effective¬ 
ness  and  efficiency  towards  the  desired  goah 

The  personnel  implementing  such  a  plan  at  each  level  - -development , 
promulgation,  execution  and  auditing  must  have  skill,  abilities,  experi¬ 
ence,  requisite  training  and  the  desire  to  carry  out  their  part.  They 
must  understand  the  necessity  of  doing  their  functions  in  accordance 
with  the  plan.  There  must  be,  however,  the  provision  for  them  to 
suggest  and  have  implemented  their  proposals  for  improvements.  This 
is  the  incentive  to  maintaining  desire  and  much  of  the  iterative  improve¬ 
ments  in  a  plan  should  come  from  the  personnel  involved  in  promulgation, 
execution  and  auditing  to  be  fed  back  to  the  developer  of  the  plan. 
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Discipline,  then,  is  a  combination  of  a  plan  and  personnel.  It  is 
a  state  of  order  maintained  by  training  and  control. 

Such  a  state  does  not  exist  within  the  DoD  today.  Because  there  is 
no  discernible  plan  (because  the  plan  or  plans,  if  any,  have  not  been 
promulgated  well)  good  execution  and  auditing  become  improbable. 

The  present  state  of  products  in  use  in  the  DoD  is  the  result  of  a 
fine  American  industry,  strong  military  department  effort  in  the  past 
and  the  happy  interplay  and  understanding  which  has  gone  on  between 
military  and  private  activities  in  the  past.  The  OSD  headquarters  efforts 
to  reduce  costs  and  improve  effectiveness  have  been  impressive  and 
have  without  question  produced  beneficial  results.  They  have  not,  how¬ 
ever,  come  as  a  result  of  a  plan  for  applying  resources  to  the  many 
disciplines  involved  in  product  contouring,  nor  have  they  been  a  product 
of  understanding  the  importance  of  basic  integrated  engineering  and  the 
use  of  incentive,  nor  of  the  importance  of  the  "discipline"  discipline. 

OSD  concepts  need  to  be  subjected  to  the  biasing  of  the  major  contour¬ 
ing  disciplines,  QARAM,  among  others. 

The  problem  is  sufficiently  important  for  a  basic  conceptual  plan 
to  be  developed,  promulgated,  executed  and  audited  for  execution  of 
the  OSD  concepts  for  product  management. 


Recommendation 

It  is  recommended  that  OSD  study  the  matter  of  the  M  discipline" 
discipline  in  the  QARAM  field  and  develop  a  plan  for  implementing  its 
needs  in  this  area,  promulgate  the  plan,  execute  the  plan,  audit  the 
plan  and  improve  the  plan. 
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INTEGRATED  ENGINEERING 


The  "think"  phase  of  product  development  all  too  often  is  considered 
to  be  only  the  conception  and  design  of  the  product.  Nothing  could  be 
further  from  the  realization  of  the  highest  effectiveness,  lowest  cost 
product. 

The  "think"  phase  of  product  exploitation  extends  through  the  entire 
life  cycle,  from  conception  through  research  and  development,  through 
engineering,  through  procurement,  production,  storage,  maintenance, 
use  and  final  disposal.  The  think  phase  must  cover  the  entire  life  cycle. 

"Think"  and  engineering  are  reasonably  synonomous  in  product 
development;  engineering  is  an  inclusive  term  wi.ich  in  essence  says 
that  professional  personnel  with  special  skills  and  training  in  a  partic¬ 
ular  area  will  apply  their  brains  in  a  disciplined  way  to  a  specific  aspect 
of  a  product's  attributes. 

The  product  desired  is  usually  defined  by  a  series  of  descriptive 
product  attributes  such  as  size,  weight,  performance,  reliability, 
maintainability,  etc. ;  items  which  can  be  measured  at  a  specific  point 
in  time.  From  these  desired  product  attributes,  engineers  with  specific 
training,  in  structures,  mechanics,  electrical  or  electronics  specialties, 
etc.  are  able  to  develop  a  basic  product  which  meets  the  specifications. 
They,  in  meeting  the  specifications,  apply  the  standard  techniques  and 
m<  s  of  tests  with  the  engineering  discipline  in  which  they  are  involved 
requires.  Thus,  a  product  is  born  in  the  conceptual  phase  in  which  all 
of  the  various  requirements  of  the  customer  have  been  considered. 

This  basic  engineering  phase  is  that  which  produces  the  substance  of 
the  product  and  determines  whether  there  is  in  fact  c.  product  which  can 
be  made  from  the  customer's  specifications.  The  bcsic  engineering 
effort  is  that  which  determines  a  product  to  the  customer's  specifications 
and  determines  the  product  attributes,  from  whence  quality  derives. 

In  addition  to  meeting  the  product  attribute  through  basic  engineering, 
there  exists  the  necessity  to  apply  contouring  techniques  which  have  not 
yet  reached,  perhaps,  the  same  validity  and  credibility  as  the  basic 
engineering  techniques  and  disciplines  but  nonetheless,  through  their 
growth,  have  provided  additional  tools  with  which  the  product  can  be 
shaped  to  maximum  effectiveness.  These  contouring  techniques  require 
that  a  product  developed  in  basic  engineering  be  locked  at  by  a  successive 
series  of  different  biases  to  obtain  the  best  possible  product.  These 
successive  biased  looks  need  to  be  taken  in  an  engineering  environment; 
they  need  to  be  taken  in  close  time  and  geographical  proximity  to  the 
basic  engineering  development  of  the  product. 
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Unfortunately,  within  the  DoD  a  proliferation  of  the  "splintered 
engineering  disciplines"  techniques  has  resulted  from  special  disciplines 
receiving  undue  emphasis,  or  having  received  proper  emphasis,  were 
allowed  to  continue  operations  external  to  the  basic  engineering  develop¬ 
ment  of  the  product  desired.  These  special  splintered  engineering 
disciplines  are  separately  organized  to  a  large  extent  and  maintain 
their  allegiance  and  their  existence  to  the  present  time  outside  the 
engineering  community.  In  the  OSD  there  is  no  basic  engineering 
capability;  in  the  OSD,  however,  there  are  large  numbers  of  persons, 
including  engineers,  involved  in  promoting  splintered  disciplines  such 
as  safety  engineering,  value  engineering,  human  engineering,  standards 
engineering,  standardization,  etc.  All  of  these  contouring  disciplines 
are  necessary;  all  of  them  have  their  place;  all  of  them  are  important 
to  the  best  final  product.  Unfortunately,  however,  with  the  type  of 
organization  which  exists  in  the  OSD,  and  is  therefore  mirrored  within 
the  military  departments,  the  emphasis  is  misplaced  often.  The 
resources  are  often  poorly  applied  and  since  there  is  no  overall  OSD 
basic  engineering  capability  and  little  overall  OSD  management  of  the 
resource  allocation  within  the  headquarters,  the  splintered  discipline 
activity  which  is  most  persuasive,  or  has  the  best  brochuremanship, 
wins  additional  resources  at  the  expense  of  activities  which  need  them 
far  more.  Further,  and  most  unfortunate  of  all,  some  of  these  splinter 
disciplines  survive  long  after  the  problem  is  no  longer  acute  or  long 
after  their  usefulness  is  ended. 

A  plan  is  needed  by  which  the  contouring  disciplines  are  effectively 
put  into  their  proper  place  in  engineering  development  where  the 
resources  which  are  applied  can  be  applied  by  a  predetermined  plan, 
where  the  resources  which  are  to  be  maintained  within  that  splinter 
discipline  are  subject  to  continuous  review. 

There  is  a  need  for  the  people  currently  involved  in  promoting 
splintered  disciplines  to  apply  their  capability  within  the  integrated 
engineering  framework.  Many  of  these  capabilities  are  not  currently 
fully  utilized  and  could  be  better  I'tilized  in  or  in  conjunction  with  other 
engineering  disciplines  and  contouring  disciplines.  Were  they  to  be 
reintegrated  into  the  basic  structure  from  which  hey  emerged,  the 
product  would  be  better  defined,  the  product  would  be  better  developed, 
built,  and  utilized. 

There  is  a  great  need  to  reduce  to  a  minimum  the  splintered 
engineering  discipline  effect  in  the  DoD,  to  establish  emphasis  on 
certain  contouring  disciplines  from  time  to  time,  but  to  insure  that 
this  emphasis  be  reduced  when  the  problem  has  been  reduced  in  scope 
and  to  end  up  with  fully  integrated  engineering  of  the  product.  Such  an 
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approach  would  eliminate  the  continuing  pressures  and  poor  engineering 
resulting  from  over -emphasizing  certain  aspects  of  the  product.  Such 
an  approach  would  result  in  the  best  possible  design  for  a  product,  the 
best  possible  plan  for  its  development,  acquisition,  production,  main¬ 
tenance,  storage,  and  use.  Such  an  approach  would  lead  to  substantially 
reduced  costs  and  substantially  better  utilization  of  the  manpower  assets 
within  the  DoD. 

The  "think"  portion  of  the  QARAM  discipline,  among  others,  needs 
to  be  included  in  the  basic  engineering  effort  for  optimizing  the  product 
attributes  for  the  product's  entire  life  cycle;  these  attributes  must  be 
contoured  by  QARAM. 

The  "do" --enforcement- -portion  of  QARAM  is  provided  by  applica¬ 
tion  of  resources,  organizational  planning  - -above  all,  it,  QARAM,  must 
be  fully  identified  and  auditable  through  the  entire  life  cycle. 


Recommendation 

It  is  recommended  that  DoD  reintegrate  the  current  splintered 
engineering  disciplines  which  are  separately  organized  and  supported 
into  a  basic  engineering  structure  and  that  as  new  contouring  disciplines 
come  along,  they  be  emphasized  as  necessary  by  management  action, 
but  that  they  not  be  allowed  to  go  out  of  the  basic  engineering  structure 
in  the  DoD. 
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INCENTIVE 


1 .  General  Discussion 

Incentive  is  a  vital  and  driving  force  wnich  encourages  man  to 
produce  his  best.  It  is  currently  not  utilized  very  effectively,  and  often 
not  at  all,  in  the  OSD  QARAM  directive  system  and  in  its  application  by 
field  activities. 

There  is  a  need  to  consider  incentive  at  every  organizational 
element.  For  example,  in  the  OSD  itself,  what  is  the  incentive  to  an 
individual  to  engineer  himself  out  of  a  job?  What  is  the  incentive  for 
him  to  do  what  is  difficult  and  contrary  to  the  desires  of  his  well- 
intentioned  but  mistaken  boss?  What  is  the  incentive  for  him  to  per¬ 
severe  in  the  face  of  conventional  rejection  of  his  idea?  What  is  the 
incentive  to  the  OSD  headquarters  as  a  whole  to  de-Parkinson  itself 
and  to  prevent  proliferation  of  people  which  has  occurred  while  simul¬ 
taneously  the  number  of  units  managed  was  reducing  itself?  Currently 
there  is  little  or  no  incentive  to  prevent  this  from  occurring.  There 
needs  to  be.  Application  of  incentive  techniques  would  provide  ways  of 
overcoming  this  particular  problem. 

In  field  organizations,  what  is  the  incentive  to  persevere  in  produc¬ 
ing  a  vitally  needed  product  for  which  the  specifications  have  been  poorly 
drawn  by  a  senior  organization?  What  is  the  incentive  to  change  an 
operating  mode  which  has  been  prescribed  by  a  senior  and  has  been 
proven  to  be  ineffective?  What  is  the  incentive  to  eliminate  unnecessary 
mechanics'  work  at  the  cost  of  additional  overhead  type  functions?  Field 
activities  have  not  been  studied  from  the  incentive  viewpoint.  The  appli¬ 
cation  of  incentive  to  field  activities  would  produce  significant  results, 
but  also  significant  changes  in  methodology,  organization,  software,  etc. 
In  this  connection,  industrial  funding  concepts  for  Government  activities 
offer  a  major  possibility  fo.  incentive  application  and  provide  a  fine 
vehicle  for  institution  of  incentive  applications. 

In  the  contractor's  area,  many  incentives  are  needed  and  could  be 
applied.  Where  specifications,  for  example,  are  set  too  high  and  cannot 
be  met  by  any  vendor,  the  vendor  who  tries  the  least  often  is  paid  the 
same  as  the  vendor  who  tries  the  hardest.  What  incentive  is  there  for 
the  vendor  to  give  the  highest  possible  approach  to  a  difficult  specifica¬ 
tion?  What  incentive  is  there  for  a  vendor  to  advise  a  Government 
activity  of  similar  products  being  made  for  other  Government  activities 
whose  specifications  are  such  that  reductions  in  cost  could  be  made? 

How  about  the  incentive  to  a  contractor  to  advise  where  other  Govern¬ 
ment  contracts  for  a  similar  product  have  had  value  engineering  or 
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money  saving  clauses  applied?  How  about  the  incentive  of  reducing  the 
specification  requirement  where  the  vendor  is  convinced  that  the  prod¬ 
uct  will  still  bt  fully  adequate  for  the  intended  service?  There  are 
many  ways  to  provide  incentives  to  the  contractor  to  do  a  better  job;  by 
applying  the  right  kinds  and  right  amounts  of  incentives,  we  can  get  his 
entire  work  force  thinking  about  our  problem.  Application  of  more 
thought  to  our  problem,  acquisition  of  more  experience  through  incen¬ 
tive  would  certainly  provide  an  improved  product  or  a  reduced  cost 
product.  Further,  could  we  provide  incentives  by  reducing  the  number  ' 
of  audits  for  a  high  quality  vendor  who  consistently  remains  in  the  high 
quality  range?  Could  we  provide  him  with  a  way  of  showing  his  contin¬ 
ued  high  quality,  thus  reducing  the  number  of  audits  of  his  process  and 
his  books?  Could  we  provide  him  with  a  rating  which  he  could  advertise 
and  give  him  more  incentive? 

As  we  discuss  incentive,  we  are  also  discussing  the  negative  of 
incentive,  i.e.  ,  penalty.  Penalty  has  not  been  adequately  utilized 
within  the  DoD  in  QARAM.  Penalties  are  sometimes  assessed  for 
lack  of  performance  of  the  product,  lack  of  timeliness,  but  they  are 
seldom  sufficient  to  overcome  the  disadvantages  to  the  Government 
which  these  deficiencies  have  incurred.  Lack  of  QARAM  is  seldom 
penalized  sufficiently.  There  has  been  little  systematic  consideration 
of  the  importance  of  these  elements. 

It  appears  that  a  study  of  penalties  for  nonperformance  in  QARAM, 
as  well  as  for  nonperformance  in  time  and  cost,  could  produce  impor¬ 
tant  gains. 

Incenave  motivation  must  consider  not  only  incentive  and  its  nega¬ 
tive,  penalty,  but  also  its  neuter,  disincentive  and  its  negative  neuter, 
dispenalty.  These  latter  two  occur  as  the  result  of  poor  policy,  poor 
communication,  etc.  ,  producing  an  effect  opposite  to  that  intended;  for 
example,  conflicting  directives  make  it  possible  to  carry  out  the  least 
demanding,  an  unattainable  specification  makes  the  small  effort  of  one 
sharp  vendor  equal  to  the  large  effort  of  an  interested  vendor  if  both 
miss  the  mark,  etc.  Disincentive  and  dispenalty  must  be  wiped  out  if 
we  are  to  acquire  QARAM  through  incentives. 


Recommendation 

It  is  recommended  that  DoD  reemphasize  incentive  as  a  motivating 
discipline,  to  be  emphasized  over  the  next  two  years  in  the  QARAM 
disciplines . 
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2.  Examples  of  Incentives 

Some  examples  of  th^  application  of  incentives  are  as  follows: 

a.  Customer  Funding  of  DCAS  Offices.  DCAS,  an  important 
field  activity  of  the  DoD  which  services  all  military  departments  and 
other  Government  agencies,  is  currently  funded  directly  /rom  its 
budgeting  and  appropriations  structure.  In  most  cases  the  customer, 
who  really  determines  how  much  work  DCAS  has  and  what  DCAS  needs 
to  do,  does  not  reimburse  DCAS  for  the  cost  of  this  function  except  for 
a  departmental  assessment.  It  appears  that  such  a  scheme  can  only 
result  in  major  difficulties  for  DCAS  in  terms  of  supporting  its  opera¬ 
tions  and  predicting  what  will  be  required  and  responding  to  the  needs 
of  the  individual  activities.  To  give  DCAS  some  incentive  beyond  that 
imposed  by  higher  authority  in  its  own  chain  of  command,  i.  e.  ,  the 
senior  DCAS  offices  and  the  OSD,  to  give  DCAS  some  incentive  to 
respond  to  the  customer,  it  would  appear  sound  to  require  that  all 
customers  fund  directly  all  of  the  services  which  they  request  of  the 
DCAS.  By  such  a  funding  arrangement,  the  customer  is  able  to  deter¬ 
mine  how  much  effort  he  wishes  to  have  expended  by  the  DCAS  and  the 
DCAS  also  is  able  to  determine  how  much  effort  is  funded  by  the  cus¬ 
tomer. 

DCAS  offices  are  currently  funded  directly  by  allotment  using 
appropriations  accounting  structure.  The  statistics  upon  which  oper¬ 
ating  allotments  are  provided  are  often,  as  is  well  known,  behind  the 
need  or  the  technology  or  often  based  on  the  ability  of  the  individual 
activity  to  be  heard  in  the  halls  of  the  budgeteers,  and,  all  too  often, 
involve  personalities  of  the  individuals  making  the  pleas  and  those 
making  the  determinations.  Such  a  scheme  is  satisfactory  perhaps 
within  a  closely  confined  area  such  as  the  OSD  headquarters  or  military 
headquarters  in  Washington,  but  is  completely  unsatisfactory  in  a  nation¬ 
wide  effort  to  cover  the  entire  field  of  production.  Under  such  circum¬ 
stances,  there  needs  to  be  an  incentive  to  the  DCAS  to  provide  the 
requisite  kinds  of  services  requested  by  the  customer,  but  even  more 
importantly,  to  provide  the  DCAS  with  a  source  of  funds  which  match 
the  work  which  he  is  requested  to  do.  Industrial  funding  meets  all  of 
these  requirements  and  it  appears  that  industrial  funding  would,  in 
addition,  automatically  provide  for  the  distribution  of  excess  DCAS 
assets  from  one  office  to  another.  Such  a  course  of  action  presu,  poses 
the  relaxation  of  conflicting  administrative  constraints  such  as  personnel 
ceilings,  overtime  restrictions ,  travel  limitations,  etc. 
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Recommendation 


It  is  recommended  that  using  activities  pay  DCAS  for  service 
rendered;  a  suggested  mode  is  through  use  of  the  industrially  funded 
concept  for  DCASR  offices  in  conjunction  with  the  relaxation  of  con¬ 
flicting  administrative  constraints. 

b.  Use  of  Industrial  Funding  Technique.  Organizations  pro¬ 
ducing  material  or  having  an  important  maintenance  or  storage  function 
can  seldom  obtain  good  information  as  to  cost  of  these  various  functions 
because  of  the  peculiar  kinds  of  accounting  which  are  forced  upon  them 
and  the  financial  support  which  is  made  available  to  them,  not  by  prod¬ 
uct,  not  by  technical  disciplines,  but  rather  in  accordance  with  an 
appropriation  structure  which  is  geared  to  the  need  of  the  legislative 
body  and  the  budget  directors  and  executors  in  the  Washington  head- 
qu  .rters.  Until  an  accounting  system  is  implemented  which  does  pro¬ 
vide  for  adequate  striation  of  charges  so  that  costs  of  QARAM  and 
other  technical  disciplines  can  be  obtained,  significant  progress  will 
not  be  made.  Major  gains  could  be  made  by  adopting  the  industrially 
funding  concept,  not  only  for  naval  shipyards  where  it  has  been  in 
existence  for  fifteen  years,  but  also  for  all  types  of  project  organiza¬ 
tions  and  field  organizations  whose  primary  function  is  to  produce  or 
store  materials.  Industrial  funding  techniques  provide  major  incentive 
to  managers  by  providing  them  with  resources  to  match  the  needs  of 
the  customers  and  providing  them  with  costs  to  show  wherein  the  money 
is  being  spent. 


Recommendation 

It  is  recommended  that  OSD  provide  for  maximum  utilization  of  the 
industrial  funding  technique  as  an  incentive  device  for  field  managers. 

c.  Funding  Guidelines.  There  is  at  the  present  time  no 
guidance  provided  to  field  activities  nor  to  their  field  headquarters  as 
to  operating  percentages  of  funds  which  should  be  devoted  to  QARAM. 
This  difficult  question  has  not  really  been  addressed  by  any  of  the  multi¬ 
tudinous  DoD  directives,  handbooks,  etc.  Indeed  it  is  not  a  question 
which  can  be  answered  by  flat  percentage.  However,  by  describing 
and  categorizing  products,  it  is  possible  that  operating  ranges  for 
QARAM  could  be  established  and  if  satisfactory  accounting  systems 
were  simultaneously  developed,  percentages  to  be  applied  to  QARAM 
would  fall  within  a  reasonable  range  for  certain  kinds  of  products. 

Some  of  the  DCAS  operations  have  generated  information  on  DCAS 
quality  costs  which  could  be  included  in  such  an  effort.  Were  these 
percentages  or  goals  to  be  established,  there  would  be  incentive  for 
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managers  to  find  ways  to  improve  their  operations,  either  quality-wise 
or  cost-wise.  At  present  there  is  no  such  incentive  since  QARAM  is, 
by  direction,  virtually  entirely  an  overhead  function  and  is  not  separated 
out  in  costs,  and  in  many  areas,  is  not  sufficiently  emphasized. 


Recommendation 

It  is  recommended  that  where  industrially  funded  operations  are 
not  feasible,  OSD  establish  the  operating  ranges  for  funds  devoted  to 
QARAM  as  an  incentive  to  field  managers. 

3 .  Government  Field  Activities 

There  exist  within  the  Government  many  incentive  programs. 
These  are  generally  departmental  in  nature  and  do  not  survive  well  the 
transmission  crossover  from  one  department  to  another  (Beneficial 
Suggestion  Program,  etc.  ).  Similarly,  specification  revision  programs 
appear  effective  intradepartmental,  but  relatively  ineffectual  in  an 
interdepartmental  sense.  Since  interdepartmental  effort  is  sharply 
increasing,  this  aspect  to  incentive  programs  must  be  emphasized. 

In  the  QARAM  field  the  interplay  between  the  DCS,  DCAS  (DCASRS), 
military  departments  and  military  field  units  makes  such  action  highly 
desirable. 

As  the  nation's  products  become  mere  complex  and  as  professional 
and  technical  societies  become  better  organized,  the  major  industrial 
concerns  in  the  United  States  are  being  brought  closer  together  in  terms 
of  QARAM.  There  are  many  audits  of  vendor  organizations  by  the 
Government  activities.  There  are  also  audits  of  vendors'  operations  by 
commercial  activities,  for  example,  in  the  automobile  industry  and  the 
aircraft  industry.  Little  use  has  been  made  by  the  Government  of  the 
vendor  audits  which  have  been  made  by  industry  for  commercial  prod¬ 
ucts;  much  use  could  be  made  of  such  audits.  It  would  be  necessary, 
of  course,  to  transfer  such  audit  information  from  the  substance  format 
set  up  by  the  commercial  activity  to  that  required  by  the  Government. 

A  simple  transposition  from  one  form  to  another  is  all  that  is  required 
in  some  cases.  Substantial  savings  could  be  obtained  by  the  Government 
in  reduced  effort  and  substantial  savings  would  accrue  to  the  vendor  were 
commercial  activity  product  audits  utilized  where  applicable.  Ihis 
incentive  to  both  Government  and  vendor  in  terms  of  reduced  costs  and 
reduced  time  could  be  substantial  in  the  complex  industry  area. 

Further,  and  perhaps  more  important,  many  vendors  are  subcon¬ 
tractors  to  several  prime  vendors,  and  provide  material  to  many 
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Government  sources.  These  vendors  are  in  a  constant  state  of  audit 
often  with  many  contractor  auditing  teams  in  plant  at  the  same  time. 
The  resulting  interference,  confusion  and  cost  is  not  in  the  best 
interest  of  the  Government,  since  the  Government  pays  the  costs  of 
audit  eventually.  Clearly,  reduction  in  such  audits  by  Government 
action  would  be  in  the  Government's  interest.  Some  reversion  to 
Government  certification  might  be  desirable.  In  any  event,  a  thorough 
study  of  this  situation  is  necessary. 


R  ecommendations 


It  is  recommended: 

1.  That  OSD  review  incentive  programs  emphasizing  their 
interdepartmental  elements, 

2.  That  Government  contracting  and  overseeing  agencies  use 
commercial  activity  audits  of  vendors  insofar  as  practical.  Further, 
that  a  study  be  made  by  OSD  to  determine  ways  in  which  replicative  or 
simultaneous  audits  of  a  vendor  for  the  same  purpose  could  be  reduced. 

4.  Vendors 

While  value  engineering  concepts  are  well  understood  in 
Government  and  industry,  some  of  the  other  incentive  modes  are  not 
particularly  well  understood.  One  of  these,  for  example,  is  the 
similar  product  savings  concept.  Vendors  often  receive  requirements 
from  Government  activities  for  products  whose  attributes  are  quite 
similar  in  most  respects  to  products  already  ordered  by  other  Govern¬ 
ment  activities  or  in  use  in  commercial  areas.  While  there  is  currently 
a  pre-award  survey  requirement  for  "similar  product"  review,  after 
the  survey  there  is  no  incentive  for  the  vendor  to  go  back  to  the  purchas¬ 
ing  agency  and  advise  that  he  has  a  similar  product  set  up  for  production 
which,  if  production  run  is  increased,  could  result  in  substantial  savings 
to  the  Government,  and  possible  higher  profit  to  the  vendor  because  of 
reduced  management  attention,  reduced  overhead  on  the  separate  and 
different  products,  etc.  There  is  no  provision  currently  for  any  savings 
of  this  type  to  be  passed  on  to  the  vendor  and  since  there  is  no  incentive 
to  him,  seldom  will  he  or  can  he  take  the  time  and  effort  to  advise  the 
contracting  agency  that  his  already  produced  product  is  adequate.  It 
would  therefore  appear  prudent  to  consider  a  "similar  product  clause" 
in  contracts  which  in  essence  would  call  for  the  contractor  to  provide  to 
the  contracting  agency  advice  as  to  similar  products  which  he  has  under 
production  or  has  produced  which  could  serve  the  same  purpose  intended 
by  the  contracting  agency  and  at  a  considerably  lower  cost.  Such 
savings,  should  the  substitution  be  accepted,  could  be  split  on  a  formula 
basis. 
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The  opposite  end  of  the  incentive  spectrum  is  penally.  Penalties 
are  well  understood  in  industry,  and  in  law,  but  they  are  not  well 
implemented  in  Government  contracting  procedures,  particularly 
penalties  for  failure  to  meet  QARAM  provisions  of  contracts.  The 
reason  seems  to  involve  in  specifications  which  are  not  clear,  or 
first-time  products,  areas  in  which  the  Government  of  necessity  makes 
changes,  etc.  Nonetheless,  there  can  be  better  utilization  of  penalties 
as  an  incentive  to  meet  specifications.  Certain  Government  agencies 
require  contractors  to  pay  re -inspection  costs  and/or  costs  incurred 
when  Government  inspectors  are  called  out,  but  the  contractor  is  not 
ready.  There  needs  to  be  added  penalty  for  failure  to  meet  QARAM 
requirements  where  such  failure  results  in  substantial  added  costs  in 
the  final  product,  (for  example,  where  the  element  purchased  is  an 
element  of  a  sub-assembly  and  the  total  sub-assembly  is  degraded  by 
the  unsatisfactory  product  received  from  the  vendor).  These  penalties 
will  be  hard  to  determine  and  enforce,  but  can  and  should  be  attempted. 

In  all  penalty  assessment  QARAM  personnel  play  a  large  role  and 
must  be  required  to  formally  provide  deficiency  information  to  Govern¬ 
ment  contracting  and  contract  administration  personnel. 


Recommendations 


It  is  recommended: 

1.  That  the  "similar  product"  savings  approach  to  providing 
material  be  studied  by  OSD  and  where  applicable  be  included  in  stand¬ 
ardized  contract  forms  with  a  formula  split  savings  clause. 

2.  That  OSD  enforce  the  application  of  penalties  provided  for 
failure  to  hneet  specifications,  particularly  in  the  QARAM  area. 

5.  Career  Patterns  in  OSD 

Career  patterns  for  personnel  in  the  QARAM  areas  do  not 
currently  exist  in  the  normal  sense  since  this  technology  is  relatively 
new.  There  are  great  needs,  there  are  insufficiently  allocated 
resources.  The  resource  application  is  being  hampered  by  lack  of  an 
adequate  career  pattern,  indeed  by  a  lack  of  adequate  information  as 
to  what  are  the  QARAM  personnel  requirements  of  the  various  military 
departments  and  the  DoD.  For  a  career  pattern  to  be  developed  which 
has  significance  there  needs  to  be  experience;  field,  headqi  arters, 
vendor- -the  complete  gamut  of  experience,  there  needs  to  be  refresh¬ 
ening  of  such  experience.  Career  patterns  for  QARAM  discipline 
people  are  important  and  could  result  in  substantially  better  utilization 
of  the  insufficient  resources  in  this  area  which  we  currently  have. 
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The  possibility  of  interchange  of  Government  and  industry  QARAM 
personnel  jshould  be  studied  as  should  the  possibility  of  advanced  degree 
study  during  a  sabbatical  year.  Career  patterns  for  QARAM  personnel 
should  be  developed  to  provide  incentive. 


Recommendation 

It  is  recommended  that  OSD  study  suitable  career  patterns  to 
provide  additional  incentive  for  DoD  QARAM  personnel. 

6.  Personal  Accountability 

The  elimination  of  personal  accountability  in  DoD  organizations 
is  working  to  the  disadvantage  of  DoD --it  is  a  disincentive  to  the  individ¬ 
ual.  For  example,  the  old  method  of  identification  of  the  inspector  by 
name  or  number  was  a  positive  incentive  through  personal  accountability. 
It  should  be  restored.  (The  DoD  Acceptance  and  Inspection  Stamp  could 
be  modified  to  provide  such.  )  Similarly,  major  efforts  should  be  made 
to  develop  concepts  and  clans  for  personal  accountability  throughout 
the  QARAM  discipline. 


Recommendation 

It  is  recommended  that  DoD  require  personal  accountability  in  the 
QARAM  discipline  as  an  incentive  to  improved  performance  and  an  aid 
to  auditability. 

7.  Disincentive 

Perhaps  the  most  discouraging  portion  of  the  entire  operation 
is  the  inadvertent  use  by  all  echelons  in  the  DoD  and  its  components  of 
disincentive  and  i'cs  opposite,  dispenalty.  Disincentive  continually  saps 
the  organization  of  the  vitality  which  is  brought  by  enthusiasm,  interest, 
and  proper  use  of  incentive.  There  are  many,  many  indications  of 
disincentive  throughout  all  operations.  For  example,  the  overemphasis 
of  certain  elements  and  certain  techniques  by  DoD  directive  or  depart¬ 
mental  directive.  Currently,  for  example,  the  very  important  value 
engineering  concept  is  being  debased  by  unrealistic  goals  and  over¬ 
emphasis,  leading  people  into  shoddy  practice.  Again,  since  the  head¬ 
quarters  value  engineering  personnel  involved  are  now  looking  for  more 
things  to  do,  they  are  spreading  out  and  aggrandizing  unto  themselves 
other  engineering  elements.  The  same  is  true  of  Zero  Defects,  a  ve**y 
valuable  program  which  has  been  prostituted  by  the  improper  use  of 
words  and  the  improper  application  of  the  term  "Zero  Defects"  to  people 
who  merely  say  they  will  try  to  do  better.  Disincentive  here  is 
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considerable.  Of  perhaps  more  demoralizing  effect  i<*  an  area  :n  which 
the  responsibility  for  turning  out  a  satisfactory  product,  or  usir.g  it,  has 
been  laid  upon  an  activity  and  simultaneously  many  directives,  by 
function,  by  money  accounting  or  by  personnel  accounting  remove  from 
the  activity  the  basic  capability  of  doing  the  job  properly.  The  continued 
diversion  of  more  resources  to  the  Washington  scene  is  often  a  disin- 
c  cntive  to  field  activities  who  do  produce  in  essence  the  entire  product 
of  the  DoD. 

There  is  considerable  evidence  of  dispenalty,  too,  in  our  acquisi¬ 
tion  operations.  The  vendor  who  produces  a  product  which  is  very 
close  to  the  specifications  often  receives  the  same  penalty  as  the  vendor 
who  produces  a  product  which  is  far  below  the  specifications.  Both 
have  missed  the  mark;  therefore,  both  receive  the  penalties.  The 
vendor  who  has  a  strong  QARAM  organization,  good  records  and  knows 
ail  of  the  details  of  his  product  often  receives  more  specification  depar¬ 
ture  notices  than  does  the  vendor  who  knows  nothing  about  his  product 
and  has  no  records.  Dispenalty  continues  to  result  in  opportunities  for 
the  marginal  and  sometimes  unscrupulous  vendor  to  operate  to  the 
detriment  of  the  tot  1  program.  We  need  to  consider  the  matter  of 
disincentive  and  dispenalty  seriously  and  remove  it  from  our  opera¬ 
tions.  Dispenalty  and  disincentive  are  the  neuter  of  incentive;  they 
can  produce  nothing --they  can  only  give  us  difficulties. 


Recommendation 

It  is  recommended  that  the  disincentive  and  dispenalty  aspects  of 
operations  be  removed  as  they  are  discovered  by  field  activities  and 
the  DoD. 

8.  General  Incentives  in  DoD 

The  need  for  attention  to  the  incentive  concept  and  approach  is 
not  restricted  to  QARAM.  As  a  rt  suit,  we  feel  a  need  to  discuss  this 
subject  in  a  broader  context. 

a.  OSD.  There  is  currently  no  incentive  within  the  DoD, 
either  at  headquarters  or  in  its  field  activities,  for  a  job  to  be  elim¬ 
inated  by  the  incumbent.  While  there  is  some  incentive  for  some  levels 
of  management  to  eliminate  jobs,  particularly  for  those  organizations 
which  are  industrially  funded,  there  is  no  incentive  currently  provided 
to  the  individual  to  engineer  himself  out  of  a  job.  On  the  contrary, 
every  influence  works  the  other  way.  Civil  Service  grade  structure 
requirements  often  relate  to  numbers  of  people  supervised,  thus, 
usually  tending  to  build  up  an  empire.  Further,  since  the  individual  has 
no  place  to  go  should  he  engineer  himself  out  of  a  job,  he  is  concerned 
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about  making  a  living  after  moving  along.  It  appears  that  there  could 
be  substantial  incentive  for  job  elimination,  however,  if  management 
would  address  itself  to  the  problem.  There  exists  currently  as  a 
result  of  the  base  closure  system,  a  major  DoD  referral  system  of 
jobs  at  all  levels.  This  referral  system  will  of  necessity  be  kept  going 
for  many  years  because  of  the  phaseout  times  for  the  various  activities 
designated  for  closure.  The  use  of  this  referral  system  and  the  freez¬ 
ing  of  certain  job  opportunities  in  it  would  offer  the  high  probability 
that  any  individual  who  engineered  himself  out  of  a  job  could  have  a 
similar  job,  or  a  job  which  would  utilize  his  talents,  available  to  him 
within  a  relatively  short  time.  Further,  since  the  problems  that  exist 
are  almost  always  greater  than  the  resources  which  we  have,  there 
could  be  set  up  a  pool  where  people  who  engineer  themselves  out  of 
jobs  could  be  utilized  on  specific  problems  pending  acquisition  of  job 
opportunity  which  they  might  wish  to  accept.  The  basic  problem, 
however,  is  how  to  get  a  man  to  engineer  himself  out  of  a  job.  There 
is  no  incentive  presently.  There  could  be  one.  For  example,  set  a 
year's  pay  as  the  incentive,  half  of  which  would  go  towards  paying  for 
six  months  leave  for  the  individual  to  do  as  he  wished- -vacation,  study, 
travel,  etc.  ;  the  second  six  months  would  be- set  aside  for  training  for 
a  new  job  so  that  in  effect  the  individual  would  then  have  a  sabbatical 
period  followed  by  a  retraining  for  a  new  job.  Such  incentive,  together 
with  a  centralized  referral  system  could  result  in  substantial  job  reduc¬ 
tion  by  engineering  by  the  job  incumbent.  Other  incentives,  such  as 
flat-out  case  payments,  could  be  used,  but  since  they  would  not  provide 
the  individual  with  a  re-entry  into  the  system  they  might  not  be  as 
effective.  In  any  event,  incentives  for  engineering  oneself  out  of  a 
job  are  needed  in  the  DoD. 

b.  Washington  Headquarters.  Incentive  for  military  depart¬ 
ment  headquarters  in  Washington  is  a  most  difficult  area.  It  is  diffi¬ 
cult  to  industrially  fund  such  headquarters,  it  is  difficult  to  acquire 
sufficient  resources  to  make  them  as  fully  responsive  as  they  would 
like  to  be,  it  is  difficult  to  find  measurement  techniques  which  indicate 
their  effectiveness  or  efficiency.  Therefore,  it  appears  that  special 
incentive  studies  are  needed  to  determine  how  the  Washington  military 
department  could  be  incentive  guided.  There  are  obviously  many  routine 
incentive  type  systems --the  Beneficial  Suggestion  system,  the  Value 
Engineering  system,  etc.  --but  these  all  work  at  the  technology,  the 
resources  and  few  of  them  get  to  the  heart  of  the  matter  of  what  does 
it  cost  to  provide  a  Navy  or  an  Army  or  an  Air  Force.  The  cost  effec¬ 
tiveness  studies  which  have  been  made  can  only  be  based  on  a  reason¬ 
able  number  of  assumptions;  unfortunately,  such  assumptions  are  not 
always  based,  nor  can  they  be  based,  on  what  the  next  year's  enemy 
v  ill  actually  do.  Cost  effectiveness  studies  can  only  be  a  first  approxi¬ 
mation  of  what  is  needed  and  how  funds  should  be  spent. 
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Similarly,  OSD  headquarters  does  not  operate  effectively  using 
incentives.  It  has  grown  over  th»j  years  from  a  few  hundred  to  many 
thousands  of  people.  Despite  its  growth  in  people  and  proliferation  of 
high  grade  structure,  it  has  not  yet  acquired  the  resources  nor  the 
organized  management  which  it  so  sorely  apparently  needs.  The 
resources  which  it  currently  has  are  ineffectively  spread  out  through 
many  different  organizations  and  are,  unfortunately,  often  involved  in 
power  struggles  because  of  the  poor  definitizations  of  what  is  required 
for  today  and  the  future.  There  is,  of  course,  considerable  knowledge 
of  what  has  occurred  in  the  past;  however,  this  is  not  a  complete  basis 
for  organization  of  application  of  resources. 

The  big  question  is,  how  can  the  DoD  eliminate  Parkinson's 
Disease,  of  which  OSD  headquarters  is,  perhaps,  a  typical  example? 

Are  there  techniques  for  organization,  are  there  techniques  for 
modifying  Civil  Service  rules,  are  there  techniques  for  job  descrip¬ 
tion  writing  which  would  permit  managerial  engineers  to  move  from 
place  to  place  within  the  organization?  Is  it  possible,  for  example,  to 
allow  an  enginee~  to  become  a  general  engineer  at  the  GS-14  level 
instead  of  a  mechanical  engineer  or  an  industrial  engineer  or  an 
electrical  engineer,  and  have  a  large  proportion  of  the  engineers  in 
a  common  pool  where  they  could  be  moved  from  operation  to  opera¬ 
tion  as  old  projects  are  completed  and  new  projects  are  begun?  Would 
it  be  possible  for  PL-313  personnel  to  be  moved  about  in  a  similar  way? 
Unless  some  way  of  cutting  the  knots  'which  developed  from  the  current 
Civil  Service  position  description  empire -building  operation  is  found, 
there  seems  to  be  no  course  for  the  future  but  that  the  DoD  headquarters 
grow  and  grow  and  grow  as  the  systems  overseeing  becomes  more 
complex.  It  appears  that  there  could  be  substantial  improvement  in 
the  management  of  the  DoD  if  an  "incentive  engineering"  management 
approach  were  taken,  with  resulting  recommendations  tc  modify  the 
manner  in  which  the  department  is  organized,  the  manner  in  which  it 
attacks  problems  and  the  manner  in  which  it  currently  fails  to  provide 
incentive  to  its  people  resources  to  move,  to  move  rapidly  from 
project  to  project  to  apply  resources  to  the  jobs  that  need  doing.  It  is 
a  fact  that  the  areas  which  are  best  staffed  are  those  areas  where  the 
problems  are  substantially  over.  The  areas  which  are  most  poorly 
staffed  are  those  in  which  the  problem  is  maximized  and  getting  larger. 
Something  must  be  done  to  improve  this  situation. 

c.  Field  Activities.  Field  activities,  like  the  central 
activities,  do  not  have  a  good  way  of  providing  career  patterns  for 
personnel,  do  not  have  a  good  way  to  move  personnel  about  from  one 
position  to  another  in  slightly  different  disciplines,  and,  above  all, 
management  has  few,  if  any.  incentives  to  eliminate,  temporarily  at 
least,  persons  or  functions  which  are  not  being  fully  utilized  because 
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of  its  known  difficulty  in  reacquiring  assets.  It  appears,  therefore, 
that  significant  gain  could  be  obtained  if,  in  the  field  of  QARAM, 
identification  of  individuals  with  these  capabilities  was  made,  that 
they  could  be  made  available  to  other  activities  when  their  own  activity 
does  not  have  full  use;  further,  that  career  patterns,  which  provided 
for  ladder -type  activities,  etc.  ,  would  be  provided,  and,  finally,  that 
a  de -Parkins oning  device  be  established.  Incentive  is  required  for 
management,  for  QARAM  personnel,  and  for  the  organization  in 
general. 


Recommendation 

It  is  recommended  that  DoD  make  a  study  of  the  broad  application 
of  the  incentive  approach  as  a  means  of  improving  management  effec¬ 
tiveness  and  resource  utilization. 
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ADEQUACY  OF  OPERATIONAL  REQUIREMENTS 


Many  Operational  Requirements  do  not  contain  comprehensive 
statements  from  which  realistic  requirements  can  be  derived  for  the 
attainment  of  the  necessary  quality  and  reliability. 

The  Operational  Requirement  (SOR,  QMR)  is  a  document  by  which 
the  Operational  Commander  states  the  need  for  the  development  of  a 
particular  operational  capability.  Normally,  it  follows  and  is  based  on 
the  information  in  a  Proposed  Technical  Approach  (PTA). 

The  PTA  is  a  document  prepared  for  the  Operational  Commander 
by  the  developing  command  outlining  technical  approaches  by  which  a 
particular  capability  may  be  achieved.  Proposed  technical  approaches 
provide  a  major  source  of  information  required  by  SECDEF  memo  of 
16  August  1961  concerning  possible  trade-offs  between  cost  and  per¬ 
formance,  and  between  lead  time  and  cost  for  use  in  RDT&E  planning. 

The  PTA  may  be  submitted  in  response  to  a  General  Operational  Require¬ 
ment  (GOR,  QMDO)  as  an  unsolicited  proposal  to  call  attention  to 
possibilities  for  a  warfare  system  resulting  from  advancing  technology 
or  in  response  to  a  Tentative  Specific  Operation  Requirement  (TSOR, 
DPQMR)  as  a  solicited  proposal  presenting  alternative  approaches  to 
attain  a  capability. 

The  TSOR  or  DPQMR  is  a  document  originated  by  the  operational 
commander  and  addressed  to  the  developing  command  or  bureau.  The 
TSOR  or  DPQMR  provides  increased  amplification  and  detail  concerning 
a  particular  operational  capability  need  which  was  stated  in  general 
terms  in  the  GOR  or  QMDO.  The  TSOR  or  DPQMR  is  an  official  request 
for  certain  information  required  by  the  operational  command  to  under¬ 
stand  better  the  scope  of  effort  and  resources  needed  to  achieve  a 
particular  capability.  Promulgation  of  a  TSOR  or  DPQMR  does  not 
establish  a  firm  service  requirement  and  does  not  authorize  the  com¬ 
mencement  of  a  new  development  program.  The  TSOR  or  DPQMR 
states  a  tentative  requirement  for  a  particular  capability,  identifies 
the  anticipated  or  existing  threat,  defines  those  performance  and  opera¬ 
tion  characteristic  envelopes  which  can  be  specified,  indicates  the  time 
period  in  which  it  is  envisioned  the  capability  is  needed,  and  usually 
states  the  number  of  systems  which  will  be  required. 

The  TSOR  or  DPQMR  is  a  step  toward  arriving  at  a  definition  of  a 
system;  its  characteristics;  its  development;  and  its  cost  of  procurement, 
operation,  and  maintenance.  It  is  a  means  for  comparing  the  effective¬ 
ness  of  a  proposed  system  with  alternative  methods  of  accomplishing 
similar  missions. 


61 


Recommendations 


It  is  recommended: 

1.  That  operational  commanders  should  be  encouraged  to  include 
in  their  statement  of  requirements  operational  information  that  affects 
reliability  and  maintainability  design. 

2.  That  during  the  conceptual  phase,  the  reliability  and  main¬ 
tainability  requirements  should  be  refined  by  analysis  and  study  of 
alternatives.  After  'he  SOR  or  QMR  is  promulgated,  trade-offs  should 
not  be  made  without  a  complete  analysis  and  synthesis  comparing  all 
the  alternatives  in  the  original  Proposed  Technical  Approach  and  others 
that  may  have  become  feasible  in  the  interim.  The  operations  research 
analysis  (operations  analysis,  systems  analysis,  operations  evaluation 
study)  on  which  the  Proposed  Technical  Approach  is  based  should  be 
made  available  to  engineers  for  translation  into  engineering  require¬ 
ments  suitable  for  contracting. 
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STRATIFICATION  OF  INHERENT /OPERATIONAL 
CHARACTERISTICS 


The  absence  of  adequate  degradation  data  and  its  effect  on  inherent 
engineering  characteristics  makes  it  difficult  to  establish  inherent 
characteristics  that  knowingly  will  meet  the  needs  of  operational 
deployment. 

The  proof  of  how  well  systems  and  equipments  meet  operational 
requirements  must  await  issue  to  operational  forces  and  is  at  that  time 
measured  and  reported  in  the  materiel  readiness  posture.  The  user, 
as  drafter  of  the  requirements  documents,  the  Qualitative  Materiel 
Requirements  in  the  Army  and  the  Tentative  Specific  Operational 
Requirement  in  the  Navy,  and  the  Required  Operational  Capability 
(ROC)  in  the  Air  Force,  thinks  primarily  in  terms  of  these  operational 
requirements  and  not  in  terms  of  the  pure  inherent  requirements  as 
differentiated  in  MIL-STD-721A  and  MIL-STD-778,  now  being  combined 
as  MIL-STD-721 B.  The  operational  requirement,  which  has  as  its 
baseline  the  inherent  characteristics,  is  complicated  by  the  addition  of 
all  the  factors  of  the  real  world  such  as  morale  and  individual  com¬ 
petence,  state  of  training,  efficiency  of  the  logistics  system,  the 
administrative  system  and  the  quality  of  the  production  item.  These 
complicating  factors  are  recognized  variables,  many  with  wide  devia¬ 
tions  and  most  with  unmeasured  or  unknown  central  tendencies. 

The  developer,  as  the  initial  entrant  in  the  acquisition  phase,  deals 
primarily  with  the  inherent  characteristics  of  systems  and  equipments. 
These  characteristics  can  be  predicted,  apportioned,  and  measured  in 
demonstrations  and  controlled  tests.  Those  characteristics  such  as 
inherent  availability,  reliability,  and  to  a  lesser  degree  maintainability, 
have  accepted  scientific  basis  on  which  to  predict  results  as  opposed 
to  a  now  existent  or  limited  empirical  base  for  those  variables  that  add 
up  to  the  operational  characteristics.  It  is  these  inherent  characteristics 
that  the  developing  agencies  can  consider  in  trade-offs  along  with  other 
performance  characteristics  in  a  matrix  that  includes  time  and  dollars. 
The  developer  can  trade-off  inherent  reliability  levels  by  adjusting  the 
testing  accomplished  during  development  in  favor  of  time  or  money,  or 
both,  as  an  example.  The  developer  does  not  have  the  K  factors  that 
degrade  the  inherent  characteristics  and  are  present  in  the  operational 
characteristics.  This  problem  is  real,  not  always  recognized,  and  at 
times  leads  to  misunderstandings  between  the  developer  and  the  user 
because  of  the  differences  in  the  baselines. 
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The  developer,  in  his  modeling  and  trade-offs  to  arrive  at  an 
optimum  model,  can  and  does  influence  many  variable  K  factors  by  the 
nature  of  his  design.  If  he  fails  to  recognize  that  there  is  a  Murphy’s 
Law  or  to  properly  consider  man-machine  relationships  in  human 
engineering,  he  may  produce  a  product  that  will  have  unanticipated 
failures  in  the  field  that  places  undue  burden  on  the  logistics  system 
and  in  turn  degrading  the  readiness  posture.  An  example  would  be 
dependence  upon  the  operator  to  operate  a  piece  of  equipment  at  the 
proper  speed  to  avoid  damaging  the  engine  as  opposed  to  providing  a 
machine  governor  that  would  prevent  engine  runaway.  On  the  other 
hand,  the  design  specifications  can  require  tolerances  of  fit  or 
characteristics  that  are  difficult  in  manufacturing,  increasing  the 
quality  assurance  problem,  and  in  extreme  cases  negating  field  and 
depot  repair.  These  affect  production,  procurement,  and  the  stockage 
position 

K  factors  associated  with  logistics,  administration,  and  other 
influencing  actions  need  to  be  determined  and  understood  for  systems 
and  equipments  before  the  user  and  developer  have  a  common  baseline 
for  translation  into  realistic  requirements.  Then,  trade-offs  can  be 
more  intelligently  assessed  and  decisions  made  on  alternatives.  A 
reporting  system  that  will  complement  the  K  factor  determination  is 
a  necessity. 

The  Army,  in  connection  with  HAWK  missile  system,  has  had 
some  encouraging  experience  with  isolating  mean  logistics  time  from 
inherent  time,  which  hopefully  can  be  translate  i  to  the  SAM-D  stated 
requirements. 


Recommendations 


It  is  recommended: 

1.  That  an  examination  be  made  of  existing  data  reporting  systems 
to  determine  their  capability  to  stratify  the  inherent  and  operational 
characteristics. 

2.  That  K  factors  associated  with  the  operational  use  of  selected 
combat  systems  and  equipments  be  defined. 

3.  That  the  services  establish  or  place  more  emphasis  on 
assurance  programs  that  will  minimize  the  degraoation  of  inherent 
reliability  of  the  design  during  the  production,  operational  deployment 
and  logiatics  support  of  military  systems  and  equipments. 


RELIABILITY  DEMONSTRATION  REQUIREMENTS  DURING 

CONTRACT  DEFINITION 


Reliability  demonstration  requirements  (tests  and  confidence 
levels)  are  not  adequately  defined  during  contract  definition  and  early 
development. 

The  publication  of  MIL-STD-785,  30  June  65,  for  mandatory  use 
by  all  departments  and  agencies  of  the  DoD,  provides  reliability  pro* 
grams  for  the  design,  development,  and  production  of  systems  and 
equipments.  It  furnishes  uniform  criteria  for  reliability  programs  and 
guidelines  for  the  preparation  of  detailed  specifications  and  contract 
work  statements.  At  the  conclusion  of  contract  definition  the  contractor 
should  furnish  the  government  with  a  Reliability  Program  Plan  which 
will  include  provisions  for  the  following: 

1.  Development  and  Qualification  Testing. 

A  planned  and  scheduled  program  of  functional  environmental 
testing  of  equipment  to  be  conducted  during  design  and  development 
phases  t  estimate  achieved  reliability  and  to  provide  feedback  of  data 
as  a  basis  for  making  reliability  improvements.  The  development 
testing  program  will  confirm  adequacy  of  selection  of  units  and  parts, 
determine  capabilities  and  safety  margins,  evaluate  drifts  of  component 
parameters  with  time,  establish  human  performance  operation  and 
maintenance  variability  criteria,  and  determine  failure -modes  and 
relative  failure -rates.  Where  such  data  are  not  available,  all  items 
of  the  system  determined  by  the  reliability  studies  to  have  a  significant 
bearing  on  inherent  reliability  will  be  tested  early  in  the  development 
program. 

2.  Environmental  Testing. 

A  program  for  development,  qualification,  and  acceptance 
testing  to  be  based  on  maximum  environmental  stress  conditions  as 
specified  by  the  procuring  activity.  When  such  environmental  criteria 
are  not  specified,  estimates  will  be  made  cf  environmental  factors 
from  experience  on  past  programs.  The  program  will  be  planned  to 
permit  evaluation  of  adequacy  of  design  of  equipment  for  the  expected 
conditions  in  the  use -environment  (e.  g. ,  ground  operation,  launch, 
flight,  orbit,  etc, )  and  include  such  considerations  as:  (1)  equipment 
location,  shock -mounting  and  truss-mounting;  (2)  environmental  problem 
areas  at  the  system,  set,  group,  unit,  and  part  level;  and  i)  the  effects 
of  these  problems  on  system  reliability.  Additional  testin  of  critical 
items  such  as  life  testing,  failure  mode  testing,  and  testr>  o  determine 
adequacy  of  safety  margins,  will  be  submitted  for  appr  al. 


3.  Part  Testing. 


Parts  to  be  used  in  the  equipment  will  be  assigned  a  reliability 
index,  failure  rate,  or  expected  probability  of  failure  under  the  applica¬ 
tion  of  stress  levels.  When  these  data  are  not  available,  they  will  be 
sought  through  parts  testing.  The  testing  specifications  and  reliability 
test  procedures  of  military  specifications  for  similar  parts  shall  be 
used.  Reported  measures  of  achieved  reliability  will  not  be  based  upon 
short  duration  tests  which  predominantly  measure  performance.  Where 
time  does  not  permit  adequate  testing  at  advanced  ages,  the  actual 
range  to  be  tested  will  be  stipulated.  Plans  for  maintaining  a  current 
record  of  th3  results  are  required.  The  test  data  will  be  retained  for 
a  minimum  period  of  2  years  from  completion  of  contract. 

4.  Reliability  Demonstration. 

a.  Initial  Plan.  An  initial  plan  for  demonstration  of  achieved 
reliability  at  specified  milestones,  including  planned  number  of  test 
articles,  accept/ reject  criteria,  and  the  associated  confidence  levels 
will  be  provided.  The  initial  plans  for  demonstration  of  reliability 
should  include  trade-off  curves  showing  number  of  test  articles  and 
operating  test  time  or  test  effort  versus  reliability  and  associated 
confidence  and  will  encompass  testing  at  the  system  level,  and  sub¬ 
system  or  unit  levels,  separately  and  in  combination. 

b.  Final  Plan.  The  final  plan  for  demonstrating  achieved 
reliability  will  provide  for  including  any  revisions  to  the  initial  plan, 
and  the  ground  rules  and  criteria  for  deciding  whether  a  test  shall  be 
classified  as  a  success  or  failure,  or  whether  a  test  shall  be  excluded 
due  to  invalid  data,  or  other  factors  interfering  with  established  test 
conditions.  Reliability  demonstration  plans  will  apply  all  results  of 
testing  and  operations  from  which  valid  reliability  measurement  or 
assessment  can  be  obtained.  Engineering  tests  and  analyses,  e.  g.  , 
test -to-failure  concepts,  are  appropriate  to  supplement  statistical 
reliability  test  plans.  The  milestones  at  which  contract  compliance  is 
to  be  demonstrated  shall  be  recommended.  The  time  for  the  submission 
of  the  final  plan  for  reliability  demonstration  shall  be  designated  as  an 
overall  program  milestone. 

c.  Test  Plans.  The  test  plans  contained  in  MIL -STD -781, 
when  applicable,  will  be  applied.  Ary  other  reliability  test  plans  pro¬ 
posed  shall  be  detailed  with  regard  to  sample  sizes,  duration  of  test, 
confidence  level,  conditions  of  test,  accept/reject  criteria,  etc.  Failure 
rate  sampling  plans  such  as  outlined  in  MIL-STD-690A  are  a  good 
example. 

From  the  preceding  it  is  evident  that  there  are  existing  provisions 
that  adequately  outline  .requirements  for  tests  and  demonstration  down 


66 


to  the  parts  level.  The  criteria  for  stated  reliability  requirements 
should  be  subject  to  trade-offs  by  the  contractor  during  contract  defini¬ 
tion  and  should  be  considered  as  a  step  in  the  refinement  of  require¬ 
ments. 


Recommendation 

It  is  recommended  that  the  requirements  of  MIL.-STD-785  be 
adequately  detailed  during  contract  definition  and  in  ail  succeeding 
program  phases. 
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reliabiuty/quality/cost/time/performance  trade-offs 


Standardized  procedures  for  collecting  quality  and  reliability  data, 
and  procedures  to  effect  trade-offs  with  program  costs  and  schedules 
are  inadequate. 

The  Final  Report  of  Task  Group  IV,  Cost-Effectiveness  Optimiza¬ 
tion  (summary,  conclusion  and  recommendations),  Weapon  System 
Effectiveness  Industry  Advisory  Committee  (WSELAC)  recommended 
that  the  Air  Force  should  identify  those  elements  in  cost-effectiveness 
analysis  on  which  further  research  must  be  done  to  increase  the  utility 
of  the  approach  to  system  evaluation  and  to  improve  the  tools  for  per¬ 
forming  cost-effectiveness  analyses. 

Reliability  is  recognized  along  with  other  performance  character¬ 
istics  as  effectiveness  elements  that  can  be  expressed  as  variables  in 
mathematical  models,  subject  to  trade-offs  and  optimization.  Theory 
of  cost-performance  relationship  is  known  among  System  Analysts  and 
Operations  Researchers  and  treatises  can  be  found  in  published  litera¬ 
ture  on  methodologies  and  analysis  techniques,  but  much  research  still 
needs  to  be  done  in  this  area.  In  actual  practice  the  marginal  cost 
associated  with  incremental  changes  in  reliability  is  difficult  to  compute 
and  as  a  consequence  has  not  generally  been  included  in  mathematical 
models  to  the  same  extent  as  other  performance  characteristics. 

Very  little  can  be  accomplished  without  data,  ^he  analysts  have 
been  hampered  by  the  lack  of  data  associated  with  acquisition  and  owner¬ 
ship  costs.  These  latter  costs  are  probably  the  more  important  to  any 
reliability  traae-oii.  Case  studies  have  shown  equipment  to  experience 
total  costs  of  one  order  ox  magnitude  greater  than  the  original  acquisi¬ 
tion  cost  during  the  remainder  of  the  life  cycle. 

It  is  recognized  that  reliability  built  into  equipment  at  the  time  of 
design  is  attainable  at  the  least  cost.  The  developer  is  faced  with 
providing  a  myriad  of  effectiveness  characteristics,  all  of  which  have 
a  cost  and  time  associated  with  them.  Unless  the  developer  is  aware 
of  the  incremental  costs  associated  wifh  reli  bility  he  will  give  this 
characteristic  the  back  seat  in  favor  of  other  characteristics  he  feels 
are  more  tangible  and  attainable  within  his  time/ budget.  In  essence 
he  doesn't  always  know  what  he  can  get  for  his  monev  in  reliability. 

A  major  problem  confronting  the  military  services  today  is  the  lack 
of  factual  data  on  the  reliability  buiit  into  existing  systems  and  equip¬ 
ments.  Of  the  equipment  in  the  field  today,  lictie  is  really  known  of  the 
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mean -time -to -fai  ure.  This  is  understandable  when  we  realize  that 
contracts  until  re  ently  did  not  require  quantitative  reliability  levels 
to  be  demonstrate  .  Field  data  reporting  under  the  Air  Force's  Manual 
66-1,  Maintenance  Management  and  The  Army  Equipment  Reporting 
System  (TAERS),  have  been  in  operational  use  for  several  years.  The 
reports  issued  did  not  furnish  reliability  data  for  the  systems  or  equip¬ 
ments.  Another  progressive  step  is  the  Air  Force  IROS  (Increase 
Reliability  of  Operational  Systems)  Program,  where  meaningful  reli¬ 
ability  data  is  being  acquired  by  determining  true  failures  occurring  in 
operational  systems  and  developing  K  factors  to  arrive  at  actual  oper¬ 
ating  hours.  Without  a  data  base,  comparative  analysis  between  existing 
and  proposed  equipments  and  the  determination  of  what  reliability  can 
be  reasonably  expected  in  the  future  evolution  becomes  hard  to  pin  down. 

The  data  base  is  progressively  becoming  better,  both  as  to  cost  and 
reliability  figures  of  merit.  In  time,  analysts  will  be  able  to  determine 
break-even  points,  and  cost-effective  trade-offs  will  be  more  meaningful 
in  the  reliability -quality  area. 


R  e  c  omm  endation  s 


It  is  recommended: 

1.  That  continued  research  be  sponsored  by  the  services  in 
improving  cost-effectiveness  techniques  and  modeling  which  consider 
reliability  and  quality  as  elements  of  effectiveness. 

2.  That  all  services  continue  their  efforts  to  refine  pertinent  data 
and  techniques  for  use  in  cost-effectiveness  trade-offs, 

3.  That  management  ensure  utilization  of  reliability -quality  cost- 
effectiveness  trade-offs  in  the  decision  making  process. 
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STORAGE  DEGRADATION  FACTORS  AS  DESIGN  PARAMETERS 


The  development  of  reliable  systems  and  equipments  is  hampered 
unless  storage  degradation  factors  are  available  for  use  and  application 
by  the  designer.  Current  techniques  are  inadequate. 

Reliability  of  systems  and  equipments  begins  during  the  design 
phase.  Considerations  at  this  time  include  environmental  conditions, 
mission  requirements,  storage  and  expected  life,  and  the  maintenance 
and  logistics  concept.  The  designer-.*  must  select  materials  that  will 
satisfy  the  above  conditions  and  requirements.  This  phase  within  the 
life  cycle  is  crucial  to  equipment  since  its  physical  and  performance 
characteristics  are  being  roughly  molded.  During  the  development 
period,  many  tests  are  conducted  that  must  be  correlated  to  the  mission 
and  environmental  requirements  of  the  equipment. 

The  designer  in  the  selection  of  material  that  must  withstand  chang¬ 
ing  environments  over  extended  periods  of  time  is  often  hampered  in  his 
development  test  by  the  lack  of  accelerated  test  means  or  predictions 
that  can  be  correlated  to  the  actual  matrix  experienced  by  the  equipment. 
This  absence  of  adequate  means  for  determining  in  a  reasonable  period 
the  expected  reliability  degradation  over  extended,  varying  environment 
periods  makes  the  demonstration  of  reliability  degradation  during  stor¬ 
age  contractually  undesirable.  Actual  experience  at  a  later  date  becomes 
the  first  real  indicator  of  any  reliability  degradation. 

It  appears  that  further  research  is  required  on  accelerated  reli¬ 
ability  testing  and  prediction  techniques  as  they  effect  reliability  over 
extended  storage  periods. 


Recommendations 


It  is  recommended: 

1.  That  the  services  conduct  studies  on  systems  and  equipments 
to  determine  the  causes  and  degrees  of  reliability  degradations  in  stor¬ 
age  so  that  programs  can  be  developed  to  minimize  degradation. 

2.  That  research  be  supported  to  investigate  accelerated  testing 
techniques  for  predicting  and  measuring  reliability  degradation  in 
storage. 
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INCENTIVES  TO  MINIMIZE  ADVERSE  RELIABILITY 
AND  QUALITY  TRADE-OFFS 


Incentives  to  minimize  adverse  trade-offs  which  affect  product 
quality  and  reliability  are  not  being  fully  exploited. 

The  use  of  Cost-Plus -Incentive-Fee  type  contracts  increased 
measurably  from  FY  1954  through  FY  1964.  The  present  trend  is 
towards  greater  use  of  multiple  incentive  contracts  with  particular 
emphasis  on  performance  parameters.  The  usage  of  performance 
incentive  contracts  increased  more  than  five  fold  during  the  period 
FY  1962-FY  19641  . 

Properly  structured  multiple -incentive  contracts  should  motivate 
the  contractor  to  strive  for  outstanding  results  in  all  three  incentive 
areas,  i.  e.  ,  cost,  schedule,  and  performance.  The  contract  should 
be  structured  so  that  the  contractor  can  by  achieving  the  desired 
performance  specified  by  the  customer  maximize  profits  in  all  these 
areas  and  permit  his  management  decisions  to  be  in  the  best  interest 
of  both  the  government  and  the  contractor. 

The  best  means  to  insure  that  trade-offs  that  adversely  affect 
reliability  are  minimized  is  to  make  certain  the  reliability  and  quality 
are  consideration  in  the  first  place.  If  reliability  and  quality  are  not 
on  the  shopping  list  of  prime  considerations  initially,  then  it  goes 
without  saying  that  their  chances  of  weighted  consideration  will  6e 
minimal  at  the  time  of  actual  trade-offs. 

The  structuring  of  multiple  performance  incentives.,  along  with 
time  and  cost  considerations,  is  a  recognized,  difficult  task  requiring 
both  project  oriented  personnel  and  contract  specialists.  Not  only  are 
there  pitfalls  in  the  conflicts  generated  by  different  profit  levels  asso¬ 
ciated  with  the  three  areas,  but  often  the  performance  incentives 
require  a  weighting  process  between  the  variable  factors  themselves. 
As  an  example,  decreases  in  weight  or  materiel  are  often  traded  off 
for  increases  in  speed,  with  both  possibly  having  decreasing  effects 
on  reliability.  In  some  equipments  such  as  surface-to-air  uiu.alt’ 
systems,  high  system  availability  is  of  utmost  importance,  w,th  a 
theoretical  infinite  number  of  trade-offs  possible  between  reliability 
and  maintainability.  This  points  up  the  need  for  some  control  on  an 
acceptable  level  of  reliability  as  well  as  demonstration  means  to 
determine  quantitative  parameters. 


*DoD  Incentive  Contracting  Guide  1965,  OASD  (I&L),  pg.  95. 
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In  monotonic  fee  concept1,  during  weapon  system  development, 
the  objectives  are  considered  to  be  (1)  the  achievement  of  a  high  level 
of  military  effectiveness,  and  (2)  a  low  life  cycle  cost  (cost  for 
development,  production,  operations,  logistics,  maintenance,  and 
training).  The  effectiveness  of  incentive  contracts  should  be  reflected 
in  the  results  of  contractors'  efforts  which  are  motivated  by  incentive 
fee  and  measured  in  terms  of  incremental  life  cycle  and  weapon  system 
effectiveness.  The  incentive  plan  should  reward  the  contractor  for 
lower  total  costs  and  higher  effectiveness.  The  basic  difference 
between  performance  incentives  in  this  monotonic  concept  and  the 
tradit  ional  multiple  incentives,  such  as  weight,  speed,  reliability, 
thrust,  etc.  ,  is  that  in  the  monotonic  concept,  the  fee  paid  the 
contractor  is  determined  by  complex  functional  relationships  in  a 
math  model  as  opposed  to  the  total  fee  being  the  sum  of  the  fees  for 
each  performance  parameter  in  the  traditional  multiple  incentive 
contract.  This  results  in  essentially  an  incremental  effectiveness  vs 
incremental  cost  relationship  for  trade-offs.  The  proponents  of  this 
sophisticated  incentive  structuring  consider  it  superior  in  that  it 
requires  the  contractor  to  make  decisions  that  are  favorable  to  the 
government  in  his  own  quest  for  incentive  fee. 

The  monotonic  concept  leaves  the  contractor  with  a  wide  latitude 
for  trade-offs  that  are  relatively  mission  oriented  over  the  entire  life 
cycle  and  not  just  the  development  phase,  but  like  the  multiple  incen¬ 
tive  contract  the  performance  characteristics  must  be  structured 
either  to  priority  of  achievement  or  subject  to  a  weighted  index  in  the 
initial  modeling. 

In  all  incentive  contracting,  the  government  should  be  able  to 
specify  acceptable  minimum  quantitative  reliability  and  must  be  pre¬ 
pared  to  insist  upon  a  simple  contractual  agreement  that  there  will 
be  no  fee  award  earned  unless  all  minimum  performance  character¬ 
istics  are  met.  This  provision,  in  the  final  analysis,  is  the  only 
assurance  that  other  characteristics  are  not  suboptimized  at  the 
expense  of  reliability  and  quality.  Values  above  this  minimum  should 
be  subject  to  trade-off  as  variables  in  the  effort  to  optimize  cost, 
time,  and  performance. 


'*DoD  Incentive  Contracting  Guide  1965,  OASD(IVL),  pg.  102. 
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Recommendations 


It  is  recommended: 

/ 

1.  That  incentives  be  designed  to  minimize  adverse  trade-offs 
which  affect  quality  and  reliability. 

2.  That  reliability  and  quality  incentives  be  structured  into 
contracts  where  possible. 
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PARTS  FOR  HIGH  RELIABILITY/LONG  LIFE  APPLICATIONS 


Mechanical,  electrical,  and  electronic  parts  (including  micro- 
circuits)  of  unknown  reliability  are  approved  for  use  by  designers  of 
equipments  for  system*-  requiring  high  reliability  or  long  life. 

The  inherent  reliability  and  longevity  of  many  military  and 
federal  standard  parts  are  quite  variable  and  generally  not  known. 
These  parts  are  usually  contractually  required.  Established  reli¬ 
ability  (ER)  parts  specifications  exist  for  twenty-two  high  usage  elec¬ 
tronic  part  types  whose  level  of  reliability  is  specified  and  known  to 
be  high.  Since  ER  parts  are  not  generally  contractually  required, 
they  are  infrequently  used  in  current  new  design,  resulting  in  high 
cost  and  long  delivery.  Contractually  specifying  ER  parts  in  develop¬ 
ment  contracts  would  ultimately  increase  their  availability,  lower 
costc,  shorten  delivery  times,  upgrade  the  life  characteristics  of 
the  parts  themselves,  and  reduce  maintenance  costs. 

Weapon  and  space  systems  are  rapidly  increasing  in  complexity; 
reliability  and  long  life  requirements  are  concurrently  becoming  more 
stringent.  More  stringent  reliability  and  quality  assurance  require¬ 
ments  have  helped  to  increase  the  reliability  of  many  systems. 

Marked  improvements  in  system  reliability  have  been  achieved  in 
systems,  suc.i  as  Polaris  and  Minuteman,  using  ER  parts  of  100% 
screened  parts  whose  life  characteristics  have  also  been  controlled. 


Recommendations 


It  is  recommended; 

1.  That  development  contracts  specify,  to  the  maximum  extent 
possible,  the  use  of  available  ER  grades  of  parts  in  all  critical  sub¬ 
systems  consistent  with  their  requirements. 

2.  That  100%  acceptance  screening,  including  burn-in  where 
applicable,  should  be  required  for  all  electronic  parts  not  covered 
by  ER  specifications  and  used  in  critical  subsystems.  Provisions 
should  be  made  to  accumulate  and  disseminate  the  data  generated 
from  these  tests. 

3.  That  the  ER  parts  specification  program*  be  expanded  to 
other  electrical  and  electronic  parts,  including  microcircuits.  Also, 
high  usage  mechanical  parts  should  be  studied  to  determine  the 
feasibility  of  ER  specifications  for  them. 
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4.  That  DoD  and  NASA  investigate  the  methods  of  qualifying 
microcircuits  based  on  commonality  between  different  types  of  circuits. 

5.  That  DoD  determine  the  feasibility  of  screening  procedures 
for  mechanical  parts,  similar  to  those  fc  r  electronic  parts. 

6.  That  DoD  determine  the  feasibility  of  expanding  the  use  of 
ER  parts  in  other  than  critical  subsystems  in  order  to  attain  economic 
production  runs  and  to  insure  availability. 

7.  That  preparation  of  the  proposed  DoD/NASA  Handbook  of 
Parts  Screening  and  Burn-in  Procedures  (for  electronic  parts)  be 
accelerated. 
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TIME  PHASING  OF  RELIABILITY  AND  QUALITY 
ASSURANCE  FUNCTIONS  IN  THE  PROGRAM  CYCLE 


Reliability  and  quality  assurance  functions  are  frequently  initiated 
too  late  in  the  program  cycle  resulting  in  necessary  functions  being 
omitted,  particularly  in  the  development  ph  se.  As  a  consequence, 
probability  of  mission  success,  reliability  assurance  and  cost-effective¬ 
ness  are  all  reduced. 

Phased  planning  and  phased  program  control  have  been  initiated 
for  large  development  projects  by  DoD  Directive  3200.9.  If  established 
DoD  decision  points  were  related  to  specific  reliability  and  quality 
assurance  functions  necessary  to  optimize  system  effectiveness,  proper 
timing  of  necessary  functions  would  be  greatly  enhanced.  Also,  such 
an  effort  would  facilitate  definition  and  validation  of  reliability  and 
quality  assurance  functions  for  fee  determination  on  incentive  contracts, 
which  are  frequently  employed  in  later  phases. 

Army  Missile  Command  Regulation  702-1  requires  a  Product 
Assurance  Plan  describing  work  elements,  their  time  phasing  and 
responsibility  for  each  new  missile  system.  Minimum  requirements 
for  each  element  and  typical  relationships  are  shown.  A  similar  effort 
is  underway  in  two  documents  being  prepared  by  Air  Force  Systems 
Command. 

One  document  (AFSCM  80  -X,  coordination  draft  dated  January  1966) 
relates  reliability  programs  and  functions  to  pertinent  phases;  the  other, 
in  rough  draft  stage,  (presently  titled  System  Program  Office/Program 
Office  Procurement  Quality  Assurance  -  AFSCM  74-X)  contains  quality 
assurance  functions  and  reliability  test  requirement  procedures. 

The  complementary  nature  of  reliability  and  quality  assurance 
functions  should  be  exploited  when  identifying  the  generic  steps  necessary 
at  each  phase.  For  example,  criticality  of  hardware  and  of  specific 
modes  of  operation  determined  during  the  development  phase  by  a  failure 
mode,  effects  and  criticality  analysis  are  useful  in  identifying  critical 
tests  and  specific  characteristics  to  be  verified  in  subsequent  test, 
inspection  and  operational  checkout.  Criticality  determinations  are 
also  used  in  assessing  failures  and  malfunctions,  evaluating  corrective 
action  and  determining  controls  and  other  actions  necessary  to  prevent 
repetition.  NASA  NPC  200-1 A  requires  government  QA  representatives 
to  consider  such  analyses  (and  other  reliability  data)  in  selecting  man¬ 
datory  characteristics  for  government  agency  inspection.  However, 
unless  these  analyses  are  executed  during  the  proper  phase,  they  are 
not  available  in  time  for  such  use. 
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Recommendations 


It  is  recommended: 

1.  That  the  services  develop  appropriate  guidelines  which  peimit 
relating  specific  reliability  and  quality  assurance  functions  to  each 
program  phase. 

2.  That  the  services  should  place  more  emphasis  on  making 
reliability  and  quality  assurance  functions  complementary  to  ensure 
completeness,  timeliness  and  cost  effectiveness.  Policies,  procedures 
and  related  procurement  regulations,  should  reflect  the  value  of  comple¬ 
mentary  reliability  and  quality  assurance  efforts. 
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SPECIFICATIONS  AND  SPECIFICATION  GUIDELINES 


Specifications  and  specification  preparation  guidelines  for  supply 
procurement  are  inadequate  in  many  instances. 

The  adequacy  of  specifications  used  for  military  procurement  has 
been  the  target  not  only  of  the  Congress  and  GAO  but  also  the  Depart¬ 
ment  of  Defense.  In  considering  the  adequacy  of  specifications,  two 
specific  areas  must  be  considered: 

1.  Specifications  covering  newly  developed  items  to  be  introduced 
into  the  supply  system. 

2.  Specifications  covering  items  already  in  the  supply  system. 

The  adequacy  of  a  specification  relating  to  1.  above  is  dependent 
upon: 

1.  How  precisely  the  requirements  and  design  constraints  delin¬ 
eated  by  the  SOR  are  quantified  during  the  prototype  production  and 
test  and  evaluation  phase;  and 

2.  How  realistically  can  the  quality  and  reliability  parameters 
indicated  by  service  tests  be  extrapolated  to  those  attainable  in  supply 
quantity  production. 

The  adequacy  of  a  specification  relating  to  2.  above  is  dependent 
principally  upon  the  effectiveness  of  a  system  for  updating  these  specifi¬ 
cations  in  relation  to  changes  in  the  state-of-the-art  and  the  availability 
of  technical  data  relating  thereto. 

It  is  an  established  fact  that  a  sufficient  amount  of  production 
engineering  and  service  testing  can  not  be  accomplished  on  a  vast 
number  of  end  items  or  components  during  the  FDT&E  phase  to  insure 
that  the  final  specification  is  adequate  for  competitive  supply  procure¬ 
ment  because: 

1.  The  RDTSjE  contractor  is  generally  selected  on  the  basis  of 
superior  technical  competence  and  ability  to  perform.  Accordingly, 
he  is  not  necessarily  representative  of  the  industry  which  normally 
become  the  suppliers. 

2.  The  prototype  and  service  test  quantities  involved  in  RDT&E 
do  not  represent  production  runs.  This  limits  the  amount  of  production 
engineering  that  can  be  accomplished.  Furthermore,  because  the 
prototype  and  service  test  quantities  are  generally  "engineering  models," 
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tests  may  lead  to  the  establishment  of  quality  and  reliability  levels  that 
cannot  be  attained  in  supply  procurement  other  than  through  sole  or 
limited  source  procurements  or  at  a  high  cost  or  both. 

RDT&E  activities  within  the  military  departments  are  fully  aware 
of  the  foregoing  and  accordingly,  some  have  taken  steps  to  insure  that 
all  aspects  of  the  specification  are  reviewed  by  competent  personnel 
during  the  engineering  development  and  test  phase  prior  to  recording 
the  requirements  and  design  constraints  and  quality  and  reliability  levels 
in  a  final  specifications  for  procurement  purposes. 

A  technique  proven  successful  to  ensure  adequate  specifications  for 
procuring  a  new  item  involves  production  testing  of  the  developers 
proposed  specification  prior  to  supply  procurement. 

Basically,  the  technique  involves  awarding  a  minimum  production 
run  of  the  item  to  a  representative  segment  of  the  industry.  Inadequacies 
and  misunderstandings  are  resolved  at  the  plant  level. 

Following  the  production  test,  the  proposed  specification  is  revised 
to  include  all  authorized  changes.  The  final  specification  highlights  the 
operational  requirement  trade-offs  that  must  be  made  and  accepted  for 
competitive  procurement. 

The  DoD  Technical  Logistics  Data  and  Information  Committee 
constituted  a  subcommittee  in  July  of  1963  to  assess  the  functions  and 
proper  content  of  military  specifications.  The  scope  of  the  subcommit¬ 
tee's  study  was: 

1.  To  evaluate  the  functions  served  by  specifications  and  their 
proper  management  to  serve  the  needs  of  DoD  and  industry  most  effec¬ 
tively. 

2.  Analyze  the  content  of  the  total  range  of  documentation  employed 
for  the  acquisition  or  purchase  of  system,  subsystems,  equipment,  parts, 
material  and  services  according  to  purpose  and  dollar  value. 

3.  Recommend  policy  and  conceptual  changes  relative  to  problems 
evident  from  the  above  evaluation  and  analysis. 

The  final  report  contains  recommendations  which  should  materially 
improve  specification  guidelines  when  fully  implemented. 
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Recommendation 


It  is  recommended  that  the  services  explore  the  feasibility  of 
expanding  the  use  of  specification  production  testing  techniques. 
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RESOURCES  FOR  ADVANCING  RELIABILITY  AND  QUALITY 
TECHNOLOGY,  RELIABILITY  AND  QUALITY  PROGRAMS, 
AND  RELIABILITY  DEMONSTRATION 


Tht  first  resource  requirement  concerns  research  and  investiga¬ 
tions  to  advance  R&Q  technologies.  This  will  improve  our  ability  to 
provide  more  accurate  predictions,  demonstrate  reliability  and  measure 
quality  with  a  minimum  expenditure  of  time  and  money.  Laboratory 
pioneering  is  required  to  improve  the  capability  of  making  reliability 
evaluations  with  the  desired  level  of  confidence  in  the  allotted  lead  time 
in  the  RDT&E  time  cycle. 

Such  evaluations  would  enable  logistics  people  to  optimize  their 
operations,  and  to  know  how  long  the  fielded  equipment  will  operate 
under  the  rigors  of  military  usage.  The  end  result  would  be  improved 
cost  effectiveness. 

In  spite  of  these  well-known  considerations,  resources  are 
inadequate  for  these  much-needed  research  and  study  procedures. 

DoD  requirements  and  implementing  directives  can  only  be  effective 
when  supported  with  adequate  resources  to  ensure  realization  of  the 
required  reliability  and  quality  in  development  programs.  The  root 
of  the  problem  is  the  emphasis  of  budget  managers  upon  commodity 
oriented  programs  aimed  at  the  acquisition  of  hardware.  As  a  result, 
studies  for  the  advancement  of  a  technology  such  as  reliability  have 
suffered. 

The  second  requirement  is  the  application  of  R&Q  programs  to  the 
development  of  military  equipment,  and  the  demonstration  of  the 
effectiveness  of  such  application.  Timely  identification  of  R&Q  func¬ 
tions  (see  page  76)  is  only  the  first  step  in  establishing  adequate  R&Q 
programs.  Once  established  at  the  proper  time  in  the  program  cycle, 
adequate  resources  in  dollars  and  competent  manpower  must  be  provided. 

Resources  or  money  and  manpower  for  both  areas  are  seriously 
inadequate  in  many  instances. 


Recommendation 

It  is  recommended  that  the  OSD  provide  the  necessary  resources  in 
manpower  and  funding  to  expand  efforts  for  the  advancement  of  reliability 
and  quality  technologies,  and  ensure  that  each  project  identifies  adequate 
resources  for  the  R&Q  program  and  its  proper  demonstration. 
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REPORT  OF  PANEL  3 


TITLE;  Quality  and  Reliability  Assurance  in  the  Production  Phase 
OBJECTIVE: 


To  improve  the  capability  within  the  DoD  to  establish  require¬ 
ments  for  and  to  obtain  materiel  possessing  the  necessary 

quality  and  reliability  during  the  production  phase. 

TOPICS  DISCUSSED: 

1.  Adequacy  and  Completeness  of  Technical  Documentation 
used  for  Materiel  Procurement  Purposes 

2.  The  use  of  Systems  Specifications  MIL-Q-9858 
MIL-I-45208  and  MIL-C -45662 

3.  Cost  Considerations  in  Production  Quality  Assurance 

4.  DoD  Quality  and  Reliability  Assurance  Technical 
Procedure 

5.  Utilization  of  Customer  Deficiency  Reports  to  Improve 
Product  Quality 

6.  The  Quality  Assurance  Role  in  Overall  Contractor 
Performance  Evaluation 

7.  Role  of  Incentives  and  Warranties  in  Improving  Quality 

8.  The  Role  of  Government  Quality  Assurance  in  Subcontracts 

9.  Contractor  Quality  Control  Versus  Government  Quality 
Assurance 

10.  The  Impact  of  Procuring  Activity  Product  Evaluation  on 
Administering  Agencies 

11.  Adequacy  of  Government  Control  Over  Nonconforming 
Supplies 

12.  Quality  Assurance  Response  to  Fluctuating  Workloads 
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I  1.  Quality  Requirements  and  Activity  for  Small  Dollar 
Procurements 

14.  Procurement  Quality  Assurance  interface  Problems 


84 


_  'j  ijigsw 


¥jg*SXl“«W 


PARTICIPANTS 


Chairman 


Captain  F.  G.  Scarborough,  USN,  HQS  DCAS 


Panelists 


Recorder 


Monitor 


Lt.  Col.  Albert  B.  Chealander,  USAF,  AFCMD, 

Los  Angeles 

Mr.  Robert  McKay,  HQS,  AFLC 

Mr.  A1  Garland,  Naval  Shipyard,  San  Francisco 

Mr.  Edward  L.  Johnston,  NAVWEPS  Quality 
Assurance  Office 

Mr.  E.  D.  Johnson,  DCASR,  St.  Louis 

Mr.  Rene  L.  LaBonte,  Defense  Personnel  Support 
Center 

Mr.  Morris  K.  Dyer,  MSFC,  NASA 
Mr.  John  J  Obren,  Army  Munitions  Command 
Mr.  Richard  Tiner,  HQS,  AMC 
Mr,  Martin  B.  Broude,  FSS,  GSA 

Mr.  J.  W.  Cornish,  HQS,  DCAS 

Mr.  John  J.  Riordan,  OASD  (I&L) 


ADEQUACY  AND  COMPLETENESS  OF  TECHNICAL 
DOCUMENTATION  USED  FOR  MATERIEL 
PROCUREMENT  PURPOSES 


Top  management  of  both  industry  and  government  have  repeatedly 
stressed  the  necessity  for  improving  the  technical  effectiveness,  criteria 
and  application  of  technical  documentation  incorporated  into  procure¬ 
ment  packages.  The  prime  objective  of  this  stress  has  been  to  reduce 
procurement  costs,  increase  competitive  procurement,  eliminate 
goldplating  (thereby  achieving  cost  reduction)  and  capitalize  on  the 
"state-of-the-art"  improvements.  Since  prior  observations  and 
recommendations  have  normally  been  extremely  general  in  nature,  it 
has  been  difficult  to  measure  the  degree  of  improvement  that  government 
agencies  have  achieved. 

Improvement  in  this  area  cannot  be  measured  separately  but  must 
be  evaluated  as  part  of  the  whole  system,  i.  e.  ,  final  product  quality 
and  performance  is  largely  dependent  not  only  on  the  technical  docu¬ 
mentation  itself,  but  also,  the  management,  control,  etc.  ,  exercised 
by  the  producer  and  the  government  contract  administration  practices. 
However,  improvement  in  this  area  could  contribute  significantly 
towards  achieving  the  prime  objective. 

Although  industry  has  at  times  been  extremely  critical  of  the 
inadequacy  and  incompleteness  of  technical  documentation,  they  have 
simultaneously  been  critical  of  technical  documentation  being  too 
detailed,  thereby  eliminating  the  application  of  their  engineering  and 
manufacturing  knowhow  to  reduce  costs.  From  an  overall  government 
viewpoint,  it  would  appear  that  the  objective  lies  closer  to  the  detail 
approach  with  provisions  to  capitalize  on  cost  reduction  aspects  through 
such  media  as  the  value  engineering  incentive  clause  in  following 
competitive  contracts. 

Within  the  present  technical  documentation  acquisition  structure, 
three  prime  techniques  are  in  effect: 

1.  Technical  documentation  is  generated  completely  within  a 
government  in-house  engineering  center,  with  or  without  contract 
technical  service  support,  i.  e.  ,  drafting  services,  editorial  services, 
etc. 


2.  Technical  documentation  is  generated  by  both  a  contractor  and 
a  government  in-house  engineering  center,  as  an  initial  joint  venture. 
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3.  Technical  documentation  is  generated  completely  by  a  contractor. 
From  a  utopian  government  viewpoint,  the  first  cited  technique  is  the 
most  desirable.  However,  from  a  practical  and  realistic  viewpoint, 
the  government  will  be  faced  with  the  data  acquisition  techniques  cited 
above,  including  some  variations  thereof,  and  the  problems  associated 
therewith  to  insure  technical  documentation  adequacy. 

Accordingly,  the  government  effort  to  assure  technical  documen¬ 
tation  adequacy  must  consider  the  following: 

1.  Administrative  Review 

Compliance  with  established  format  requirements  (MIL  STD-100, 
MIL-STD-1 000,  MIL  -STDs  -1  thru  31,  etc.). 

2.  Design  Review 

Technical  adequacy  of  the  design  disclosure  technical 
documentation. 

While  the  inadequacies  in  data  such  as  restrictive  legends  or 
missing  drawings,  specifications,  etc.  are  frequently  found  in  assembling 
a  procurement  package,  inadequate  QA  provisions  are  less  often  detected. 
The  pressure  to  compete  to  the  maximum  has  increased  problems  in 
this  area  since  the  first  sign  of  inadequate  QA  provisions  frequently 
comes  to  light  when  defective  product  is  found  in  the  system.  Specifi¬ 
cations  which  may  have  been  satisfactory  when  used  witn  sole  source 
producers  many  times  become  highly  inadequate  when  used  with  a  new 
source. 

One  way  of  countering  some  of  these  problems  is  to  assure  that 
quality  engineering  type  personnel  participate  in  the  development  of 
technical  documentation  and  then  later  give  ail  procurement  packages 
a  final  thorough  QA  review  prior  to  the  initiation  of  a  procurement 
action.  While  quality  specialists  are  not  a  panacea  for  all  problems, 
the  questioning  mind  of  persons  who  have  lived  with,  and  attempted  to 
accept  product  based  on  specification  requirements  frequently  raises 
questions  which  have  gone  unnoticed  in  engineering  assessments. 

Since  the  government  deals  with  a  multitude  of  :ontractors,  one  of 
the  problems  it  is  faced  with  is  that  of  communications,  i.  e.  ,  to  be  able 
to  convey  to  the  technical  contractor  staff  through  written  media  (notes 
on  a  drawing,  the  interpretation  of  words  in  a  specification,  the  meaning 
of  quality  assuranc**  contract  clauses,  etc.  )  the  total  requirements  of 
an  item.  This  is  not  meant  to  infer  that  these  technical  staffs  do  not 
possess  the  capability  but  rather  that  these  staffs  must  go  through  a 
"learning  curve"  due  to  changes  in  the  staff  brought  about  by  resignations. 


87 


retirement,  etc.  Accordingly,  improved  communications  could  be 
generated  by  centralizing  within  the  quality  assurance  organization  of 
the  following  functions: 

1.  Providing  quality  assurance  contract  clause  input  to  the  con¬ 
tracting  officers. 

2.  Arranging  for  the  conduct  of  post  award  type  conferences,  when 
required  (new  producers,  etc.  ),  to  assure  that  the  contractor  has  a 
thorough  understanding  of  the  product  quality  requirements. 

Trends  to  depend  more  and  more  on  contractor  knowledge  and  less 
on  Government  in-house  capability  to  test  and  evaluate  certainly  must 
be  considered  as  contributing  to  some  of  the  poor  quality  procurement 
packages  which  find  their  way  into  contracts. 


Recommendations 


It  is  recommended: 

1.  Short  Range  -  That  OSD  issue  policies  which  provide  for  a 
requirement  that  quality  assurance  personnel: 

a.  Review  the  technical  documentation  for  adequacy  prior  to 
a  procurement  action. 

b.  Provide  the  required  quality  assurance  contract  clause 

input. 

c.  On  major  procurement  contracts  arrange  for  the  conduct 
of  a  post  award  type  conference  to  assure  a  sound  understanding  of  the 
product  quality  requirements. 

2.  Long  Range 

a.  That  all  elements  of  DoD  continue  to  stress  the  necessity 
for  sound  "Configuration  Management'1  practices  including  adequate 
testing  and  evaluation  of  both  the  materiel  and  technical  documentation. 

b.  That  OSD  establish  a  DoD  Committee  to  determine  what 
additional  quality  assurance  contract  clauses  would  be  beneficial  for 
inclusion  in  ASPR. 

c.  That  all  elements  of  DoD  continue  to  stress  the  necessity 
for  quality  assurance  engineering  personnel  participating  in  the  develop¬ 
ment  of  technical  documentation. 
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THE  USE  OF  SYSTEMS  SPECIFICATIONS  MIL-Q-9858 
MIL-I-45203  AND  MIL-C -45662 


There  is  a  lack  of  definitive  criteria  for  the  selection  and  inclusion 
into  contracts  of  systems  specifications.  Correspondingly,  there  is  a 
lack  of  complete  guidance  for  the  equitable  evaluation  of  implemented 
specifications  between  contractors.  Compounding  this  problem  is  a 
move  by  some  industry  associations  to  introduce  supplemented  systems 
specifications  which  will  fractionate  the  standardized  approach  achieved 
in  MIL -Q -9858,  MIL -.1-45208  and  MIL-C -45662. 

Prior  to  the  issuance  of  the  'A"  ievision  to  MIL-Q-9858  there  were 
several  system  specifications,  in  the  quality  area,  being  imposed  by 
contracts  on  defense  suppliers.  The  scope  and  depth  of  coverage  varied 
depending  upon  the  military  service/purchasing  office  that  developed 
and  invoked  the  specification.  This  resulted  in  confusion  among  con¬ 
tractors  who  did  business  with  more  than  one  service  and  among  contract 
administration  offices  w'ho  may  have  received  contracts  from  purchasing 
offices  outside  their  command  by  interchange. 

Compounding  the  problem  of  multiple,  service -oriented,  systems 
specifications  was  the  problem  of  evaluation  criteria  of  contractor 
implemented  systems.  Greater  confusion  existed  in  this  area  in  that 
there  was  as  much  difference  in  evaluation  criteria  used  within  the 
services  as  there  was  between  the  services. 

Publication  of  MIL-Q-9858A,  MIL -I-45208A.  and  MIL-C-45662A 
achieved  a  degree  of  uniformity,  by  self -designation,  as  only  authorized 
specifications  on  the  subjects  covered.  The  military  services  then  took 
the  necessary  steps  to  rescind  their  own  publications.  Guidance  of  the 
methods  of  evaluating  these  programs  was  published  at  a  much  later 
date  and  in  the  ca««:  of  MIL-I-45208A  it  has  not  been  published  to  date. 
Therefore,  standa  lization  was  achieved  through  the  systems  specifica¬ 
tion,  but  evaluation  was  and  has  remained  varied. 

Presently  there  is  broad  policy  guidance  on  some  of  the  problem 
areas.  For  example,  Section  XIV  of  ASPR  contains  limited  guidance  on 
the  application  of  MIL-Q-9858  and  MIL-I-45208.  This  guidance  is  not 
specific  nor  does  it  provide  definitive  criteria  for  the  selection  and 
application  of  these  specifications.  In  fact,  there  is  a  wide  variation  in 
the  use  of  these  specifications  by  the  various  DoD  procuring  activities. 
Some  activities  apply  MIL-Q-9858  to  virtually  all  of  their  contracts, 
while  others  use  it  only  on  complex  major  end  item  or  cost  type  contracts. 
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The  same  variation  is  prevalent  in  the  application  of  MIL-I-45208.  Most 
of  the  DoD  procuring  activities  appear  to  have  inadequate  knowledge  of 
these  system  type  specifications. 

Further,  there  is  insuffici  ;nt  guidance  on  the  actions  that  should 
be  taken  when  there  is  evidence  of  a  supplier's  noncompliance  with 
system  specification  requirements.  This  particular  problem  is  further 
compounded  when  the  product  offered  to  the  Government  for  acceptance 
meets  the  product  contractual  requirements.  Furthermore,  acceptance 
of  nonconforming  materiel  by  waiver,  due  to  the  urgency  of  delivery 
requirements,  dilutes  the  QAR's  effectiveness. 

The  metal  producing  industries  (steel,  aluminum,  copper  and 
nickel)  have  developed  a  proposed  quality  program  specification 
specifically  designed  for  metal  mill  products.  This  draft  has  been 
presented  to  OSD  for  approval.  OSD  advised  the  metal  producing 
industries  that  a  new  specification  would  not  be  acceptable  but  that  a 
redrafting  of  MIL-Q-9858A  with  a  "slash  1"  would  probably  be  acceptable. 
This  type  of  fractionization  of  the  basic  quality  program  specification 
completely  defeats  the  purpose  of  "one  specification  for  a  quality 
program."  In  addition,  publication  of  such  a  product-oriented  specifica¬ 
tion  would  open  "Pandora's  Box"  to  any  number  of  "special"  quality 
specifications. 

These  problem  topics  are  directed  to  the  absence  of  definitive 
policy  guidance  and  the  nonuniformity  of  the  selection  and  application 
of  system  type  specifications  by  the  DoD  procuring  activities.  This 
lack  of  uniformity  could  result  in  a  wide  variation  in  costs  incurred  Ly 
the  Government  for  like  materiel  procured  under  the  different  types  of 
systems  specifications.  Further,  the  nonuniformity  in  selection  and 
enforcement  of  these  specifications  affects  a  supplier's  ability  to  be 
truly  competitive  in  his  bidding. 


Recommendations 


It  is  recommended: 

1.  That  OSD  develop  definitive  criteria  as  to  where  and  under 
what  circumstances  each  system  specification  should  be  imposed  by 
contract. 

2.  That  OSD  develop  and  publish  uniform  and  definitive  guidance 
for  evaluating  the  effectiveness  of  a  supplier's  implemented  program 
simultaneously  with  issuance  of  the  specifications.  This  guidance  must 
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also  include  remedial  actions  to  be  taken  when  a  contractor  fails  to 
meet  the  system  requirements  of  the  specifications, 

3.  That  OSD  direct  that  no  supplemental  specifications  are  to  be 
issued  to  the  three  existing  system  specifications. 
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COST  CONSIDERATIONS  IN  PRODUCTION  QUALITY  ASSURANCE 


The  effect  on  costs  of  imposing  fewer  or  additional  quality  require¬ 
ments,  both  on  the  part  of  the  contractor  as  well  as  the  Government, 
has  not  had  the  same  level  of  attention  as  provided  to  other  procurement 
aspects  such  as  performance  requirements  and  delivery  schedules. 

The  percentage  of  procurement  costs  associated  with  quality  is  influenced 
by  several  factors,  among  which  are  complexity  of  the  product,  state 
of  the  art,  policies  of  the  procuring  offices  and  the  policies  of  the  con¬ 
tractors.  It  is  generally  conceded  that  these  costs  can  represent  several 
percent  of  the  total  procurement.  The  lack  of  quality  cost  information 
concerning  Government  and  contractor  operations  constitutes  a  void 
with  management  process. 

Some  considerable  work  has  been  accomplished  by  individual  con¬ 
tractors,  Departments  and  professional  societies,  as  well  as  universities 
in  studying  the  costs  associated  with  quality.  Various  systems  have 
been  developed  that  would  enable  management  in  a  given  situation  to 
track  and  control  quality  costs  at  a  reasonably  optimum  level.  Some  of 
these  systems  are  very  comprehensive  and  involve  appreciable  changes 
in  accounting  systems  in  order  to  produce  the  required  information. 

With  the  revision  of  MIL-Q-9858  in  December  1963,  a  requirement 
was  introduced  calling  for  contractors  to  maintain  and  use  quality  cost 
data  as  a  management  element  of  the  quality  program.  The  specific 
quality  cost  data  to  be  maintained  and  used  is  determined  by  the  con  ■ 
tractor.  Since  the  issuance  of  this  requirement  there  has  been  little 
policy  guidance  issued  by  the  OASD  (I&L)  on  the  subject.  The  absence 
of  guidance  setting  forth  a  definition  of  these  costs  and  the  intended  use 
thereof  by  both  the  contractor  and  government  has  contributed  to  a 
general  lack  of  enforcement  of  the  requirement.  In  some  cases  where 
contractors  have  developed  cost  information,  there  exists  a  question 
as  to  the  usefulness  of  the  data. 

A  few  cases  have  been  reported  where  contractor's  quality  cost  has 
been  separately  identified  in  cost  proposals  in  advance  of  contract 
award.  This  is  an  area  that  should  be  thoroughly  evaluated  as  to 
desirability,  and  if  it  is  determined  to  be  desirable,  a  uniform  definition 
as  to  what  these  costs  consists  of  should  be  developed  and  issued  so  as 
to  assure  consistency  in  evaluation  of  contractors. 

Different  procedures  have  been  employed  by  the  Departments  and 
DSA  in  charging  contractors  for  the  government  costs  associated  with 
reinspections.  This  would  be  those  reir.spections  required  as  the  result 
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of  government  rejection  of  tendered  materiel.  Differences  in  policy 
also  exists  for  the  obtainment  of  consideration  from  a  contractor  in 
connection  with  the  acceptance  of  nonconforming  materiels.  Both  of 
these  areas  should  be  thoroughly  evaluated  to  determine  the  desirability 
of  adopting  a  single  policy  to  be  employed. 


Recommendations 


It  is  recommended: 

1.  That  the  OASD  (I&tL)  sponsor  a  study  group  to  explore  all 
facets  of  production  quality  costs  of  the  contractor  and  government. 
Based  upon  this  study,  identify  those  elements  of  costs  that  should  be 
identified  and  maintained  by  contractors  and  the  Government. 

2.  That  the  study  group  develop  a  uniform  policy  relative  to 
charging  contractors  for  the  Government  costs  associated  with  rein¬ 
spections  of  rejected  materiel.  A  policy  should  also  be  developed 
providing  for  uniform  application  of  costs  considerations  in  connection 
with  the  acceptance  of  nonconforming  materiel. 
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DoD  QUALITY  AND  RELIABILITY  ASSURANCE 
TECHNICAL  PROCEDURE 


The  dynamic  world  we  live  in  today,  which  presents  new  challenges 
to  weapon  system  developers  in  the  form  of  availability  of  new  materiels 
and  of  rapidly  changing  production  techniques,  dictates  that  our  quality 
and  reliability  assurance  programs  likewise  be  dynamic.  However, 
with  few  exceptions,  little  new  in  the  way  of  policy  or  procedure  has 
been  issued  by  OSD  in  this  area  in  the  past  several  years.  There  are 
areas  such  as  metrology  and  calibration  which  are  not  covered  by  an 
OSD  policy  other  than  guidance  on  contractor  requirements  (in  Specifica¬ 
tion  MIL -C -45662A,  dated  February  1962,  and  in  DoD  Handbook  H-52). 
There  is  a  real  need  that  a  DoD  Instruction  be  issued  outlining  quality 
assurance  policy  on  metrology,  calibration,  and  inspection  equipment, 

Further,  most  of  the  existing  OSD  policies  and  procedures  on 
procurement  quality  assurance  have  not  kept  pace  with  the  rapid  changes 
being  made  by  other  elements  of  OSD  in  the  procurement  function.  For 
example,  DoD  Handbook  H-109,  "Statistical  Procedures  for  Determining 
Validity  of  Suppliers'  Attributes  Inspection,  "  is  based  on  MIL -STD -105, 
but  MIL-STD-105  has  been  revised  four  times  since  H-109  was  published. 
Handbook  H-109,  if  updated,  would  be  a  useful  tool  for  the  uniform 
evaluation  of  a  contractor's  quality  records;  and  could  provide  a  good 
measure  to  be  included  in  development  of  quality  criteria  for  contractor 
performance  evaluation.  Similarly,  MIL-STD-109,  "Quality  Assurance 
Terms  and  Definition,  "  has  not  been  updated  to  reflect  the  latest  con¬ 
cepts  of  government  or  suppliers'  quality  responsibilities. 

The  "H"  series  of  DoD  Inspection  and  Quality  Control  Handbooks 
carry  the  designation  "Interim"  which  fosters  the  impression  that  they 
lack  permanency.  These  should  be  updated  and  published  as  official 
documents.  A  further  problem  exists  in  that  the  "H"  series  documents 
are  not  available  through  normal  distribution  channels  since  they  are 
not  official  documents  from  the  standardization  standpoint. 

There  exists  a  real  and  urgent  need  for  updating  and  taking  a  hard 
look  at  our  present  procurement  quality  assurance  concepts,  policies, 
and  procedures  to  assure  that:  (1)  they  are  mutually  compatible;  (2) 
they  are  complete  with  respect  to  all  required  and  useful  government 
procurement  quality  assurance  actions;  (3)  they  provide  definitive 
guidance  as  to  contractor  responsibility  versus  the  government  agencies' 
responsibility;  and  (4)  effective  means  are  established  to  keep  them 
current. 
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Recommendation 


It  is  recommended  that  OSD  establish  an  effective  mechanism  to 
assure  that  a  single  series  of  quality  and  reliability  assurance  policies 
and  procedures  is  officially  published,  and  is  kept  up-to-date  to 
continually  reflect  the  latest  technological  advances. 
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UTILIZATION  OF  CUSTOMER  DEFICIENCY  REPORTS  TO 
IMPROVE  PRODUCT  QUALITY 


Based  on  government  reports  and  resultant  articles  which  have 
recently  made  headline  reading  in  the  newspapers  and  other  publications, 
there  is  a  definite  indication  that  poor  quality  materiel  is  entering  the 
Government  supply  system.  This  poor  quality  materiel  can  be  basically 
divided  into  two  categories  namely,  (1)  materiel  which  has  been  fabri¬ 
cated  in  accordance  with  the  technical  documentation  specified  within 
the  contract  and  is  found  in  actual  usage  not  to  meet  customer  require¬ 
ments  and  (2)  materiel  that  has  been  accepted  by  the  Government,  which 
does  not  meet  established  technical  requirements  specified  within  the 
contract.  In  either  situation,  expeditious  receipt,  processing  and 
evaluation  of  customer  deficiency  reports  could  lead  to  improvement  in 
the  quality  of  materiel  being  fabricated. 

Although  the  expeditious  reporting  of  deficient  materiel  by  the 
customer  can  definitely  be  of  extreme  value,  the  immediate  application 
to  production  is  governed  by  the  following  factors: 

1.  The  materiel  is  still  in  production  and  the  product  deficiency 
is  due  to  the  lack  of  quality  control  on  the  part  of  the  producer  and 
quality  assurance  practices  on  the  part  of  the  government. 

2.  The  materiel  is  still  in  production  and  the  deficiency  is  one  of 
design.  Design  deficiencies  can  be  categorized  as: 

a.  A  deficiency  that  can  be  immediately  corrected  via  the 
application  of  mandatory  engineering  change. 

b.  A  deficiency  that  is  of  a  magnitude  that  requires  extensive 
engineering  effort  to  correct  the  deficiency. 

3.  The  materiel  is  presently  out  of  production,  but  future  buys  are 
contemplated. 

Based  on  the  factors  cited  above,  it  can  be  seen  that  customer 
deficiency  reports  can  definitely  be  an  asset  to  improving  product  quality 
providing  that  management  systems  are  established  for  the  expeditious 
flow  and  application  of  corrective  actions. 

The  value  of  user  data  in  improving  product  quality  depends  on 
timeliness  of  receipt  of  information  and  the  ability  to  relate  reported 
deficiencies  to  conditions  in  production.  Conditions  creating  late  receipt 
of  information  from  the  field  are  not  always  controllable  due  to  supply 
distribution  systems  and  actions  at  receiving  stations  which  have  higher 
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priority.  Inadequate  reports  probably  will  never  be  completely  resolved 
but  any  actions  that  will  encourage  better,  more  accurate  participation 
of  using  activities  should  be  tried. 

Another  area  for  further  exploration  is  DoD  wide  guidance  on 
actions  to  be  taken  on  receipt  of  this  type  information.  Failure  to  reply 
to  field  information  or  sterotyped  answers  tend  to  discourage  field 
participation.  Emphasis  should  be  placed  on  the  fact  that  deficiency 
information  is  primarily  input  data  for  communication  between  govern¬ 
ment  activities.  This  is  not  intended  to  preclude  submission  of 
deficiency  information  to  the  contractor  but  to  place  the  focal  point  for 
assuring  proper  corrective  action  with  government  activities  rather  than 
the  contractor.  More  and  closer  government  involvement  in  the  actual 
investigation  to  correct  the  deficiency  would  be  more  helpful  than  just 
limiting  participation  of  the  government  activity  to  that  of  a  referral 
agency. 

It  is  not  necessary  nor  desirable  that  one  standard  form  be  developed 
within  the  military  services  for  user  deficiency  reports,  i.  e.  ,  reports 
from  the  field -users  of  the  equipment.  The  type  of  information  desired 
and/or  reported  varies  with  the  type  of  equipment.  However,  the  existing 
complaint  type  forms  (AMC  1229,  AFTO  109,  AFTO  29,  NAVWEPS 
13070/5,  etc.  )  which  essentially  deal  with  materiel  not  meeting  specified 
contract  requirements  and  represent  transmittal  information  between 
government  agencies,  indicate  a  fruitful  area  for  form  and  procedure 
standardization.  The  information  required  to  perform  an  adequate 
investigation  on  a  production  line  is  similar  regardless  of  the  buying  or 
using  agency  involved.  As  a  minimum,  action  along  this  line  would 
reduce  problems  for  CAS  components  who  are  now  confronted  with  a 
multitude  of  forms  all  of  which  are  intended  to  perform  a  like  function. 


Recommendations 


It  is  recommended: 

1.  That  OSD  establish  a  project  for  the  development  of  a  standard 
DD  Form  and  appropriate  procedures,  which  car.  be  used  as  a  trans¬ 
mittal  sheet  for  submitting  deficiency  reports  to  CAS  components. 

2.  That  all  elements  of  DoD  review  their  present  deficiency 
reporting  systems  to  assure  that  they  provide  for  an  expeditious  flow 
of  deficiency  reports  thru  quality  assurance  channels  to  either  design 
agencies  (design  deficiencies;  or  contract  administration  activities 
(qual  ty  deficiencies),  as  appropriate. 
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THE  QUALITY  ASSURANCE  ROLE  IN  OVERALL 
CONTRACTOR  PERFORMANCE  EVALUATION 


Since  the  President's  Committee  on  Government  Contracting  for 
Research  and  Development  recommended  an  exchange  of  contractor 
evaluation  data  among  government  agencies  in  1962,  and  since  the 
issuance  of  ASPR  1  -902  and  1-903  covering  minimum  standards  for 
responsible  prospective  bidders,  considerable  work  toward  development 
of  an  effective  Contractor  Performance  Evaluation  (CPE)  Program  has 
taken  place  within  OASD  (I&L).  In  addition  to  the  overall  work  directed 
by  OSD,  CPE  programs  of  various  types  have  been  developed  and 
implemented  by  the  Military  Departments  and  DSA. 

The  primary  evaluation  factors  on  all  of  these  programs  are  cost 
and  schedule.  The  quality  aspects  of  contractor  performance  are 
mentioned  in  the  ASPR  references  above;  however,  they  are  usually 
considered  to  be  a  part  of  contractor  "technical  performance.  "  Possibly 
as  a  result  of  the  lack  of  specific  emphasis  as  a  separate  CPE  rating 
factor,  the  quality  area  has  not  been  adequately  defined  nor  has  its 
relative  importance  to  other  evaluation  factors  been  determined. 

The  lack  of  specific  identification  and  recognition  of  quality  as 
a  key  element  in  CPE  is  having  a  detrimental  effect  on  product  quality 
and  reliability,  and  therefore,  mission  capability.  The  detrimental 
effect  is  brought  about  by  the  contractor  applying  effort  to  perform  best 
in  those  key  areas  which  are  specifically  evaluated  (i.  e.  ,  cost,  schedule). 
The  magnitude  of  the  problem  will  become  more  apparent  as  the  overall 
CPE  program  is  more  fully  implemented,  unless  steps  are  immediately 
taken  to  strengthen  the  quality  area. 

An  essential  part  of  effective  and  meaningful  CPE  is  data  generated 
by  plant  level  quality  assurance  personnel  during  day-to-day  operations. 

In  order  for  full  use  to  be  made  of  such  data  in  overall  CPE  with  the 
least  amount  of  difficulty,  it  must  be  generated  through  a  DoD-wide 
quality  assurance  program  employing  common  procedures  and  common 
terms.  At  present,  several  different  quality  assurance  programs  are 
utilized  by  DoD  elements  which  employ  different  data  collection  and 
reporting  methods. 

The  fact  that  numerous  departmental  CPE  programs  are  in  use 
which  employ  different  procedures,  complicates  implementation  of  an 
overall  DoD  CPE  program.  Development  work  is  still  going  on  in  each 
of  these  programs,  with  some  further  along  than  others,  however  each 
has  strong  points  which  should  be  considered  for  use  in  the  overall 
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program.  These  programs  have  resulted  in  the  generation  of  a  con¬ 
siderable  amount  of  data,  but  due  to  limited  departmental  application, 
these  data  are  not  available  to  all  procuring  and  administration  activities 
that  could  make  use  of  them. 

The  Panel  is  aware  of  the  CPE  Pilot  Program  which  is  being 
effected  in  the  supply  and  equipment  area  by  DSA  at  the  request  of 
OASD  (I&L).  It  is  recognized  that  an  essential  element  of  an  effective 
CPE  Program  is  simplicity,  and  that  the  evaluation  factors  contained 
on  the  DSA  Form  352 -A  (TEST)  Contract  Performance  Evaluations, 
have  been  kept  to  a  minimum.  It  is  considered  however,  that  further 
study  of- these  factors  should  be  undertaken  at  the  earliest  possible  time 
in  order  to  determine  if  refinements  are  required. 


Recommendations 


It  is  recommended: 

1.  That  OSD  establish  a  task  group  composed  of  quality  assurance 
personnel  to  evaluate  the  contract  administration  quality  assurance 
programs  now  carried  out  by  the  several  DoD  elements  and  develop  a 
DoD-wide  program  for  implementation  at  the  earliest  practicable  date. 
Procedures  and  resultant  data  recording  and  reporting  methods  developed 
should  be  compatible  with  CPE  program  needs. 

2.  That  OSD  establish  a  task  group  of  quality  assurance  personnel 
to  develop  specific  quality  assurance  factors  for  inclusion  in  the  DoD 
CPE  program.  This  group  should  consider  all  factors  which  have  been 
developed  for  departmental  CPE  programs  and  those  used  in  the  DSA 
pilot  program.  Suggestions  should  be  solicited  from  industry.  Factors 
developed  should  encompass  evaluation  of  quality  program  elements  as 
well  as  product  characteristics. 

3.  That  OSD  establish  uniform  quality  considerations  for  the  DoD- 
wide  CPE  program  which  includes  the  best  features  of  departmental 
CPE  programs.  This  program  should  consider  automation  so  that 
immediate  response  can  be  made  to  queries  for  quality  and  reliability 
information  by  any  procuring  or  administering  office  during  preaward 
and  subsequent  contract  phases.  This  program  should  be  adaptable  to 
individual  contracts  as  well  as  individvial  contractors,  and  encompass 
as  many  contractors  as  possible.  The  cost  of  operation  versus  value 
received  must  be  carefully  analyzed  in  developing  the  program  scope 
and  operational  methods. 
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ROLE  OF  INCENTIVES  AND  WARRANTIES  IN 
IMPROVING  QUALITY 


DoD  has  provided  for  the  use  of  contractual  incentives  in  ASPR, 

Part  4,  Types  of  Contracts  and  ASPR,  Part  17,  Value  Engineering. 

This  guidance  has  been  supplemented  by  numerous  Defense  Procurement 
Circulars.  In  addition  to  the  above,  noncontractual  incentives  have  been 
employed  by  procurement  activities.  These  generally  take  the  form  of 
public  recognition  of  superior  performance  by  individual  programs 
through  such  media  as  the  awarding  of  flags,  plaques,  and  certificates 
of  achievement.  DoD  has  provided  a  third  type  of  incentive  in  ASPR 
1-324,  contractual  warranties. 

1.  Contractual  Incentives 

Contractual  incentives,  as  provided  for  in  ASPR,  are  extremely 
broad  in  scope.  Practically  any  aspect  of  performance  may  be  placed 
under  incentive  provisions.  These  provisions  permii  monetary  returns 
to  the  contractor  for  achievement  of  stated  goals.  The  goals  are 
usually  expressed  in  terms  of  cost  reduction,  scheduled  delivery  and 
performance  of  the  item  or  service. 

''Performance"  in  this  sense  means  mission  accomplishment;  i.  e. , 
whether  or  not  the  end  item  functions.  This  preoccupation  with  the  end 
item  masks  the  fact  that  there  should  be  methods  of  measuring  progress 
toward  the  achievement  of  the  quality  goal  during  the  production  phase. 
The  reason  for  this  consideration  is  that  quality  and  reliability  evaluation 
elements  have  not  been  well  defined  in  incentive  contracts.  A  penalty 
levied  on  the  contractor  after  failure  to  meet  a  performance  goal  is  like 
closing  the  barn  door  after  the  horse  is  stolen.  Quality  and  reliability 
incentives  should  be  extended  into  the  production  phase.  Failure  to  do 
so  up  to  this  time  is  possibly  due  to  the  difficulty  of  defining  such 
factors. 

Apart  from  performance  incentives  as  applied  to  the  area  of  major 
systems  anu  weapons,  there  are  other  areas  in  which  performance 
incentives  may  be  utilized  to  improve  quality.  For  example,  if  there 
exists  definitive  requirements  for  an  item  including  acceptable  quality 
levels  (AQL's)  should  extra  payment  be  made  for  delivery  of  product 
better  than  the  AQL?  Generally  speaking,  it  is  not  believed  that  this 
is  a  suitable  are  for  incentive  payments.  The  AQL  or  other  standard 
of  performance  imposed  by  the  contract  should  be  set  at  that  point  which 
will  provide  a  sufficiently  broad  base  for  procurement  of  a  competitive 
nature  and  yet  meet  the  needs  of  the  user.  Thus,  a  contractual  incentive 
payment  for  better  quality  might  tend  to  restrict  competitive  procurement 
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and  go  against  the  principle  that  standards  should  not  be  3et  higher  than 
required.  Nevertheless,  it  is  recognized  that  for  a  high  unit  volume 
procurement  of  a  given  item  a  small  increase  in  the  quality  of  product 
will  on  the  average  provide  a  larger  number  of  useable  units  with  a 
longer  serviceability  or  appearance  life.  For  example,  a  procurement 
for  100,  000  hydraulic  fittings  permits  delivery  of  4%  defective  product. 

If  the  actual  delivery  is  no  more  than  2%  defective,  obviously  there  are 
2,  000  more  units  that  can  be  used.  This  situation  must  take  into  con¬ 
sideration  a  contrary  factor,  namely  a  fact  that  small  increments  in 
quality  improvement  could  become  disproportionately  costly.  This 
brings  us  back  to  the  proposition  that  the  AQL  or  other  standard  of 
performance  imposed  by  the  contract  is  probably  the  best  compromise 
between  minimum  needs  and  broadly  based  competition. 

Value  Engineering  Incentives  are  designed  to  reward  contractors 
who  suggest  changes  in  design,  manufacturing  processes,  or  use  of 
newly  designed  or  other  substitute  components  which  lower  the  cost 
without  downgrading  quality  or  performance  of  the  end  item.  Since  a 
VE  change  may  improve  performance  at  lower  cost,  value  engineering 
incentives  may  be  considered  under  the  general  topic  of  incentives  as  a 
means  of  improving  quality. 

2.  Noncontractual  Incentives 

Noncontractual  incentives,  such  as  the  awarding  of  flags, 
plaques,  or  certificates  of  achievement,  recognize  superior  performance 
or,  at  least,  recognize  consistently  good  performance  and  timely  delivery 
arising  out  of  a  well-organized  management  effort.  They  are  particularly 
suitable  for  use  in  competitive  procurements  of  high  unit  volume  with 
definitive  specification  requirements  where,  as  stateu  above,  incentive 
payments  for  delivery  of  quality  better  than  required  by  the  contract  are 
inadvisable.  Noncontractual  incentives  are  designed  to  promote  improved 
management  and  use  of  quality  programs  t hat  will  better  assure  delivery 
of  conforming  product  on  schedule. 

Such  incentives  are  based  on  the  principle  that  people  will  strive  for 
recognition  of  achievement  even  though  there  may  be  no  direct  monetary 
reward  involved.  The  achievements  ox  the  Zero  Defects  program,  a- 
adopted  by  thousands  of  major  manufacturers,  attest  to  this.  Qu-  ty 
recognition  programs  are  a  significant  factor  in  assuring  delivery  of 
conforming  supplies  on  schedule. 

3.  Warranties 

Contractual  warranties,  which  provide  that  the  jontractor 
warrants  the  quality  of  product  at  time  of  delivery  to  conform  to  con¬ 
tractual  standards,  are  an  indirect  means  of  improving  quality  in  at 


they  provide  penalties  for  nonconformance.  They  may  be  considered 
as  incentives  in  reverse. 

When  the  Government  exercises  its  rights  under  warranty  provisions 
of  the  contract,  it  recognizes  the  existence  of  a  failure  in  the  contractor’s 
quality  program  as  well  as  in  the  Government's  quality  assurance  pro¬ 
gram.  The  Government's  loss  usually  cannot  be  fully  regained  by  any 
action  it  takes  under  the  warranty  article  primarily  because  it  has  been 
denied  the  use  of  acceptable  product  during  the  interval  from  date  of 
original  delivery  of  nonconforming  product  to  the  data  of  fruition 
involving  actions  of  replacement,  repair  or  recovery  of  monetary  com¬ 
pensation  for  damages  suffered  at  the  time  of  delivery.  If  the  defective 
product  has  been  discovered  through  normal  government  procurement 
quality  assurance  actions  prior  to  acceptance  resulting  in  rejection  of 
product,  any  delays  in  delivery  of  conforming  product  occasioned  by 
the  rejection  may  be  compensated  for  by  an  adjustment  of  contract  price 
in  consideration  for  a  possible  extension  of  delivery  schedule. 


Recommendations 

It  is  recomn.’ended: 

1.  That  OSD  request  the  Services  and  DSA  to  furnish  the  results 
of  their  experience  to  date  with  incentives  and  warranties,  relative  to 
quality  and  reliability. 

2.  That  OSD  identify  quality  and  reliability  evaluation  elements 
during  production  for  performance  incentive  contracts. 

3.  That  OSD  assess  the  overall  impact  of  warranties  with  emphasis 
in  the  considerations  of  their  cost,  their  relationship  to  all  types  of 
incentives  and  their  appropriate  application. 

4.  That  OSD  use  the  data  for  developing  further  guidelines  for  the 
use  of  procuring  activities. 
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THE  ROLE  OF  GOVERNMENT  QUALITY  ASSURANCE 

IN  SUBCONTRACTS 


The  DoD  manages  over  half  of  the  federal  budget;  more  than  fifty 
billion  dollars  annually.  At  the  present  rate,  about  thirty  billions  of 
that  go  into  subcontracts.  Usually  major  system  procurements  require 
the  contractor  to  subcontract  some  portion  of  the  work.  The  spread 
may  include  the  largest  "prime"  plants  at  one  end  and  the  smallest  of 
shops  at  the  other.  Subcontract  business  is  not  limited  to  the  small 
"bucket"  covered  by  the  non-resident  quality  assurance  representative. 

The  lack  of  subcontractor  conformance  to  quality  requirements  of 
critical  characteristics  in  items  procured  by  prime  contractors  con¬ 
tinues  to  be  a  serious  and  costly  problem  to  the  government.  Many 
defective  components  and  parts  have  been  installed  in  major  weapon 
and  space  systems  as  a  result  of  this  problem.  Government  attempts 
at  assuring  adequate  quality  control  of  subcontracted  supplies  has 
resulted  m  imposing  large  source  inspection  workloads  upon  government 
quality  assurance  personnel  servicing  subcontractor  plants. 

Why  Govermnent  Source  Inspection?  The  requirements  for  govern¬ 
ment  inspection  are  spelled  out  in  the  Armed  Services  Procurement 
Regulations  (ASPR)  and  are  written  into  government  contracts.  These 
inspections  are  to  assure  contract  compliance  and  to  protect  the  interests 
of  the  government.  Stated  as  an  oversimplification,  the  Government 
inspects  certain  selected  important  characteristics,  performs  acceptance 
inspection  and  accepts  those  supplies  and  services  that  are  found 
satisfactory.  Often  the  individual  making  acceptance  for  the  Govern¬ 
ment  cannot  assure  all  of  the  necessary  tests  and  examinations  on 
subcontracted  supplies.  He  therefore  may  request  assistance  from  the 
government  agency  having  cognizance  over  the  subcontractor.  Requests 
for  such  assistance  are  initiated  as  statements  calling  for  government 
source  inspection  (GSI)  on  purchase  orders  and  are  usually  supported 
by  a  letter  of  delegation  from  the  requesting  agency. 

Government  source  inspection  should  not  be  requested  on  items 
which  can  be  adequately  inspected  upon  receipt  at  the  prime  contractor’s 
plant.  GSI  is  usually  requested  on  shipments  direct  from  subcontractors 
to  government  installations.  On  noncomplex  items,  the  purchase  orders 
may  require  certificate  acceptance  or  acceptance  at  destination. 

Government  quality  assurance  at  the  subcontractor's  plant  is 
essentially  an  extension  of  government  quality  assurance  at  the  prime 
contractor's  facility.  Some  problems  encountered  by  government  quality 


103 


assurance  personnel  at  subcontractor  plants  are  late  receipt  of  the 
subcontract  or  letter  requesting  GSI,  insufficient  information  in  the 
subcontract  or  the  letter  requesting  GSI,  lack  of  adequate  prime/sub- 
contractor  quality  assurance,  and  the  large  number  and  variety  of 
plants  to  be  covered  on  an  itinerant  basis. 

The  current  policy  of  "disengagement'1  insofar  as  GST  is  concerned 
has  many  different  interpretations.  It  was  obviously  designed  to  dis¬ 
courage  GSI  and  to  place  the  burden  where  it  belongs,  on  the  prime 
contractor.  The  prime  contractor  is  solely  and  exclusively  responsible 
for  the  quality  of  all  goods  and  services  delivered  to  the  government, 
whether  the  goods  and  services  are  produced  in  the  contractor's  plant 
or  procured  from  his  suppliers. 

It  is  well  established  that  the  Government  reserves  the  right  to 
verify  contract  compliance  by  performing  inspection  at  selected  points 
in  the  process  including  government  source  inspection  under  subcontracts. 
Some  government  quality  assurance  activities  have  eliminated  the 
practice  of  requesting  GSI  on  prime  contractor  purchase  orders.  Others 
request  GSI  only  for  those  selected  items  possessing  critical  character¬ 
istics  which  cannot  be  verified  upon  receipt.  Still  others  continue  to 
request  GSI  without  regard  to  the  complexity  or  criticality  of  the  sub¬ 
contracted  items.  In  some  instances,  prime  contractors  encourage 
GSI,  with  a  view  toward  reducing  their  control  at  the  subcontractor 
plant  or  in  their  receiving  inspection,  or  a  combination  of  both. 

Inspections  performed  by  the  Government  are  essential  elements 
of  a  verification  system  and  not  necessarily  a  duplication  of  the  con¬ 
tractor's  effort.  The  goal  of  such  a  verification  system  is  to  take 
sufficient  samples,  but  only  sufficient  samples  to  determine  product 
quality  on  which  to  base  reasonable,  valid  decisions  for  acceptance. 

The  system  strives  to  be  both  effective  and  economical.  The  contractor 
is  paid  to  control  the  product,  processes  and  systems.  Government 
source  inspection  verifies  that  control  was  established  and  product 
quality  maintained. 


Recommendation 

It  is  recommended  that  OSD  incorporate  the  following  concepts  in 
policy  and  procedural  directives  in  order  to  clearly  define  the  Govern¬ 
ment  quality  assurance  role  in  subcontracting: 
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a.  Hold  prime  contractors  completely  responsible  for  the 
quality  of  supplies  received  from  their  subcontractors  and  vendors 
through  clear  and  definitive  contractual  language,  notwithstanding  GSI. 

b.  Require  government  quality  assurance  review  of  prime 
contractor  proposals  for  subcontracting. 

c.  Strengthen  government  review  of  prime  contractor's  quality 
system  including  written  procedures  for  control  over  the  quality  of 
subcontracted  supplies. 

d.  Define  the  specific  circumstances  under  which  GSI  may 
be  applied 

e.  Provide  that  all  requests  for  GSI  contain  absolute  identifi¬ 
cation  of  specific  products,  processes,  critical  characteristics  and 
other  requirements  to  be  examined. 
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CONTRACTOR  QUALITY  CONTROL  VERSUS  GOVERNMENT 

QUALITY  ASSURANCE 


Procedures  developed  and  used  by  the  Departments  and  DSA  for  the 
conduct  of  procurement  quality  assurance  contain  some  basic  conceptual 
differences.  In  the  product  verification  area,  for  example,  some  of 
these  procedures  emphasize  the  utilization  of  standard  sampling  plans, 
where  the  sample  sizes  to  be  verified  by  the  Government  equals  or 
approaches  the  sample  sizes  employed  by  the  contractor.  Provisions 
are  included  for  reducing  sample  sizes  and  frequency  of  sampling 
based  upon  contractor -demonstrated  effectiveness.  Other  procedures 
emphasize  the  measurement  of  the  contractor's  effectiveness  by  the 
employment  of  a  constant  sample  size  at  an  established  minimum 
frequency  that  provides  for  increasing  the  amount  of  Government  verifi¬ 
cation  contingent  upon  the  contractor's  demonstrated  effectiveness.  In 
addition  to  this  constant  sample  size  technique,  provisions  are  made  for 
direct  product  verification  for  acceptance  purposes  which  allow  for 
recognition  of  purchasing  office  requirements  as  well  as  local  deter¬ 
mination  as  to  necessity. 

Variations  also  exist  between  the  Departments  and  DSA  procedures 
in  the  approach  to  be  followed  in  determining  that  a  contractor  is 
following  his  prescribed  procedures. 

These  conceptual  differences  between  the  Departments  and  DSA 
are  particularly  evident  in  the  product  verification  area.  These 
variances  appear  to  reflect  differences  in  philosophy  as  concerns  the 
purpose  for  conducting  product  verification.  On  one  side,  emphasis  is 
placed  on  product  verification  for  product  acceptance  purposes,  whereas 
on  the  other  side  product  verification  is  accomplished  primarily  for 
the  purpose  of  determining  contractor's  effectiveness  in  controlling 
product  quality,  which  in  turn  provides  the  basis  for  product  acceptance. 

There  is  a  need  for  a  greater  degree  of  standardization  by  the 
Departments  and  DSA  in  the  procedures  to  be  utilized  for  determining 
that  a  contractor  is  effecting  the  required  degree  of  product  quality 
control  in  accordance  with  the  terms  of  the  contract.  It  is  considered 
absolutely  necessary  that  the  procedures  provide  recognition  of  the 
relative  importance  of  individual  procurements  and  a  clear  distinction 
between  the  activities  associated  with: 

1.  Review  of  contractor's  written  quality  control  procedures. 

2.  Determining  on  a  continuing  basis  contractor's  adherence  to 
his  written  procedures. 
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3.  Direct  government  product  examination  for  product  acceptance 
purposes. 

4.  Evaluation  of  the  accuracy  of  contractor's  decision  regarding 
product  quality. 

5.  Corrective  action. 

These  procedures  must  take  into  consideration  the  basic  DoD  policy 
setting  forth  the  contractor's  responsibility  for  controlling  product 
quality  and  must  recognize  the  economical  significance  as  concerns 
employment  of  government  quality  assurance  personnel.  It  is  equally 
important  that  these  procedures,  primarily  as  they  involve  product 
verification,  provide  for  flexible  applications  at  the  local  level.  They 
should  be  developed  to  allow  for  increased  or  decreased  Government 
effort  commensurate  with  demonstrated  contractor  effectiveness. 

It  is  the  opinion  of  the  panel  that  the  proposed  Appendix  Q  to  the 
ASPR  as  presently  developed  by  the  ASPR  Subcommittee  does  not  fully 
meet  these  requirements. 


Recommendation 

It  is  recommended  that  the  ASPR  Committee  assure  the  development 
of  standardized  procurement  quality  assurance  procedures  to  be 
employed  by  the  Departments  and  DSA;  that  these  procedures  reflect 
the  basic  policies  of  the  DoD  concerning  the  contractor's  responsibility 
for  the  control  of  product  quality,  and  recognize  their  impact  on 
Government  quality  assurance  manpower.  Procedures  should  be 
adaptable  to  different  products  and  to  allow  for  increased  or  decreased 
government  effort  commensurate  with  demonstrated  contractor  effective¬ 
ness.  Recognition  should  be  provided  to  the  intrinsic  value  of  individual 
products  to  the  total  military  mission.  Considering  the  relative 
importance  of  these  procedures,  they  should  be  adequately  service- 
tested  and  evaluated  prior  to  full  implementation. 
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THE  IMPACT  OF  PROCURING  ACTIVITY  PRODUCT 
EVALUATION  ON  ADMINISTERING  AGENCIES 


Basic  to  good  management  is  an  ability  to  assess  efficiency  or 
effectiveness  of  the  operation  being  managed.  The  DoD  in  recognition 
of  this,  has  imposed  upon  the  departments  requirements  for  evaluation 
of  the  quality  of  material  entering  or  in  the  inventory  (DODI  4155.  11). 
The  military  departments  have  all  established  methods  and  procedures 
to  control  and  evaluate  the  quality  of  material  entering  the  inventory. 

The  methods  used,  the  extent  of  coverage,  the  points  in  time  where 
the  evaluation  is  made  all  vary  greatly  from  activity  to  activity.  In 
many  cases  the  evaluation  included  detailed  involvement  in  the  adminis¬ 
tration  of  the  contract  in  the  plant.  The  establishment  of  the  Defense 
Contract  Administration  Service  and  corresponding  changes  in  depart¬ 
mental  responsibility  under  Project  60  created  changes  of  considerable 
magnitude  within  all  departments.  High  on  the  list  of  changes  in  many 
activities  was  the  need  to  re-evaluate  the  methods  used  in  assuring 
quality  of  product  under  the  new  concepts  of  operation. 

Some  of  the  differences  in  evaluation  techniques  are  due  to  com¬ 
modity  peculiarities  which  experience  has  shown  are  necessary.  Others 
are  due  to  practices  which  grew  up  with  the  agency  and  which  require 
considerable  personnel  reorientation  to  completely  resolve.  Every 
effort  is  being  made  to  revise  and  develop  internal  regulations  com¬ 
patible  with  departmental  responsibility  for  the  quality  of  materiel  and 
responsibilities  of  the  Contract  Administration  Services  (CAS)  for 
in-plant  quality  assurance.  However,  complete  reorientation  of 
personnel  involved  is  difficult  and  has  not  been  accomplished  as  rapidly 
as  desired.  For  example,  some  of  these  "people"  problems  between 
DoD  agencies  have  been  brought  about  by  key  inspector,  commodity 
specialist  or  technical  representative  visits  during  production  and  that 
have  unnecessarily  been  escalated  to  DoD  levels  due  to  misunderstand¬ 
ings.  These  problems  are  receiving  appropriate  attention  in  the  depart¬ 
ments  and  exp  ‘rience  has  shown  that  resolution  is  possible  at  the 
operating  levels  once  there  is  a  thorough  understanding  by  all  parties 
of  departmental  responsibility  and  approach.  The  problems  that  have 
developed  have  been  primarily  due  to  previous  concepts  being  carried 
over  into  a  new  environment.  The  panel  concludes  that  there  is  no 
significant  impact  on  the  administering  agency  when  procuring  agency 
evaluations  are  properly  conducted. 
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Recommendations 


It  is  recommended: 

1.  That  DoD  personnel  make  every  effort  to  keep  interface 
problems  occurring  due  to  procuring  agency  product  evaluations  at  the 
operating  levels.  With  proper  understanding,  "people"  problems 
resulting  from  Project  60  reorganization  difficulties  can  be  resolved 
without  escalation  to  higher  management. 

2.  That  no  action  be  taken  by  OSD  at  this  time  to  standardize  the 
procuring  activities'  approach  to  product  evaluation. 
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ADEQUACY  OF  GOVERNMENT  CONTROL  OVER 
NONCONFORMING  SUPPLIES 


Nonconforming  supplies,  as  discussed  herein,  contain  minor 
departures  from  the  contract,  specification,  drawing  or  other  applicable 
requirements  as  opposed  to  major  departures  which  require  contract 
change  or  other  equitable  consideration.  The  primary  purpose  for 
government  participation  in  the  control  of  nonconforming  supplies  is  to 
preclude  acceptance  of  any  nonconformance  that  will  adversely  affect 
equipment  performance,  reliability  or  maintainability.  The  acceptance 
of  any  nonconforming  supplies  is  a  privilege  extended  by  the  government 
to  avoid  unnecessary  waste  of  materials,  time,  or  money. 

Some  DoD  contract  administration  activities  have  established 
Material  Review  Boards  to  formalize  government  control.  The  Material 
Review  Board  generally  consists  of  one  government  and  two  contractor 
personnel  with  selected  skills  and  specialties  that  are  closely  associated 
with  the  type  of  product  or  processes  involved.  The  right  of  disapproval 
is  vested  in  each  member  since  unanimous  approval  is  required  for 
acceptance.  Usually,  only  material  accepted  by  both  contractor  members 
is  presented  to  the  government  member. 

Before  presenting  material  to  the  board,  the  contractor  is  required 
to  review  nonconformances  in  detail  and  assure  that  material  is  not 
presented  if  it  can  be  reworked  or  completed  to  be  entirely  conforming, 
or  if  it  is  economically  unuseable  and  must  be  scrapped.  Depending 
upon  local  option,  material  that  can  be  used  as  is,  or  that  can  be  made 
functional  by  application  of  standard  repairs  previously  approved  by 
the  board,  may  be  presented  to  the  board  or  accepted  by  the  contractor 
without  board  action.  Thus  subsequent  acceptance  of  nonconforming 
material  into  the  system  based  on  the  precedent  is  established.  Criteria 
established  on  the  basis  of  precedent  acceptance  may  also  be  used  by 
the  formal  Material  Review  Board  in  the  disposition  of  nonconforming 
supplies,  this  time  with  concurrence  of  the  government  representative. 
Acceptance  based  on  precedent  dilutes  requirements  for  effective 
corrective  action.  Adequate  corrective  action  measures  are  required 
for  presentation  to  the  board  when  considering  the  nonconforming 
material  acceptance.  Too  frequently  the  corrective  action  proposed  is 
not  adequate  or  the  board  fails  to  monitor  the  effectiveness  of  corrective 
action  implementation. 

The  procedures  used  by  the  various  DoD  components  vary  in  practice 
and  intensity.  Some  contractors  have  been  required  to  conform  to  more 
chan  one  government  MRB  procedure.  This  in  effect  has  produced  more 
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than  one  quality  standard  even  in  the  same  plant,  often  for  like  or 
similar  items  being  procured.  These  variations  have  contributed  to  an 
indifferent  attitude  by  many  contractors,  poor  motivation  for  effective 
corrective  action,  and  failure  to  notify  local  government  personnel  when 
items  containing  nonconforming  materials  are  presented  for  acceptance. 

Some  contractors  interpret  the  material  review  procedure  as  a 
method  of  authorizing  raw  material  substitutions  when  they  should  be 
following  Class  I  change  or  waiver  procedures.  When  this  interpreta¬ 
tion  is  concurred  in  by  government  personnel,  a  proliferation  of  raw 
material  substitutions  may  follow.  Some  of  the  more  obvious  results 
of  raw  material  substitutions  are  that  they: 

1.  Permit  deviations  from  contract  requirements. 

2.  Ignore  the  requirement  to  eliminate  the  cause. 

3.  Ignore  the  requirement  for  corrective  action. 

4.  Circumvent  the  established  method  for  processing  Class  I 
changes  or  waivers. 

5.  Facilitate  the  use  of  contractor  excess  inventory  materials. 

Although  results  "1"  through  "5"  above  may  be  undesirable  in  most 
applications,  there  are  occasions  when  it  is  not  economically  feasible 
to  revise  old  drawings,  with  little  future  use  potential,  to  replace  obso¬ 
lete  materials  with  currently  available  items.  Result  "5"  above  is 
frequently  most  desirable  to  make  maximum  use  of  our  national  resources 
and  to  reduce  procurement  costs.  Typical  reasons  presented  for  sub¬ 
stitution  of  raw  materials  are  "unable  to  procure,  "  "material  in  stock,  " 
or  "parts  urgently  needed  in  production.  " 

Because  of  a  lack  of  clearly -defined  policy  and  procedural  guidance 
to  control  nonconforming  supplies  of  Class  II  or  variation  category,  the 
causes  for  generating  nonconforming  supplies  are  not  being  corrected. 
Thus,  in  many  instances,  nonconforming  material  is  being  accepted  as 
a  general  practice.  In  addition,  time  and  schedule  constraints  often 
encourage  acceptance  of  nonconforming  items  based  on  precedents 
established  by  previous  "use  as  is"  acceptance  decisions.  In  most 
Government  acceptances  of  nonconforming  material,  no  consideration 
from  the  contractor  is  demanded  by  tne  Government. 

It  is  vital  that  the  primary  DoD  objective  for  controlling  noncon¬ 
forming  supplies  include  the  elimination  of  the  causes  of  recurring 
discrepancies  and  to  prevent  the  occurrence  of  similar  discrepancies. 
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A  DoD  panel  with  quality  assurance  field  representation  should  be 
established  to  prepare  policy  and  procedural  guidance  for  use  by  all 
DoD  agencies.  Such  guidance  should: 

1.  State  clearly  the  requirements  for  contractor  control  of  non- 
conforming  supplies. 

2.  State  clearly  the  requirements  for  government  control  of  non- 
conforming  supplies  in  the  contract  administration  environment. 

3.  Require  effective  corrective  action  as  a  condition  of  acceptance 
for  each  nonconformance. 

4.  Limit  acceptance  of  nonconforming  items  to  those  in  process 
at  the  time  the  original  discrepancy  is  discovered. 

5.  State  clearly  that  substitution  of  raw  materials  and  parts  shall 
not  adversely  affect  end  item  performance,  reliability,  or  interchange- 
ability. 


Recommendation 

It  is  recommended  that  the  OSD  develop  and  publish  uniform  policy 
and  procedural  guidance  for  contractor  and  government  control  over 
nonconforming  supplies  and  materials. 
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QUALITY  ASSURANCE  RESPONSE  TO  FLUCTUATING  WORKLOADS 


While  all  procurement  management  echelons  experience  problems 
in  responding  effectively  to  fluctuating  workloads,  the  most  heavily 
impacted  organizations  are  those  engaged  in  contract  administrative 
functions,  particularly  those  involved  in  a  geographical  operation.  The 
manpower  of  these  organizations  is  tied  to  a  "level  funding"  concept, 
budgeted  in  advance;  whereas,  the  workload  is  generated  by  immediate 
changes  in  procurement  priorities  and  quantities. 

From  a  practical  standpoint,  these  organizations  have  had  to  project 
manpower  increases  and  decreases  essentially  on  the  basis  of  the  total 
procurement  budget.  In  a  vast  majority  of  cases  they  are  not  made 
aware  of  new  workloads  in  advance  of  the  pre -award  phase.  The  time 
lapse  between  pre -award  and  award  of  contract  does  not  allow  for 
justifying  additional  manpower  recruitment  and  training. 

In  any  sizeable  organization  involved  in  a  geographical  quality 
assurance  activity,  it  is  generally  possible  to  adjust  manpower  to  cover 
minor  increases  or  decreases  in  workload.  However,  these  actions 
are  obviously  limited. 

A  significant  increase  in  procurement  quality  personnel  within  an 
organization  based  upon  fluctuating  workloads  can  have  an  adverse 
effect  on  the  organization  in  a  declining  workload  situation.  The  place¬ 
ment  of  temporary  employees  is  one  solution  to  the  resultant  adverse 
effects  of  a  build  up  in  the  organization.  However,  this  route  does  not 
eliminate  the  problems  associated  with  obtaining  additional  manpower, 
obtaining  necessary  budget  support,  identifying  and  recruiting  qualified 
personnel  and  provide  for  the  required  training,  all  in  a  timely  manner. 

A  possible  solution  to  the  problem  of  immediate  response  to 
fluctuating  workloads  may  lie  in  the  CAS  organizations  being  funded  for 
direct  labor  support  from  the  procurement  programs  generating  the 
workload  requirement.  This  would  permit  the  manpower  authorizations 
to  fluctuate  with  the  workloads. 

Civil  service  procedures  involving  the  hiring  of  temporary  personnel 
are  essentially  no  different  than  those  required  to  be  followed  for  the 
employment  of  permanent  personnel.  The  mechanism  for  identifying 
readily  available  qualified  personnel  to  be  employed  on  either  a  perma¬ 
nent  or  temporary  basis  should  be  reviewed  to  determine  what  steps 
could  be  taken  to  speed  up  the  process. 


Another  area  that  hampers  the  ability  of  these  organizations  to 
adjust  to  fluctuating  workloads  is  the  amount  and  apparent  inconsistent 
applications  of  mandatory  inspections  directed  by  the  procuring 
activities.  A  review  of  the  data  contained  in  Table  I  concerning  the 
expenditure  of  directed  mandatory  inspection  by  government  inspection 
activities  reveals  that  for  the  five -month  period  from  December  1965 
to  April  1966  a  23  percent  increase  (percentages  computed  on  the  basis 
of  total  manhours  reported)  in  directed  mandatory  inspection  activity 
was  effected.  During  the  same  period  of  time,  there  was  only  a  10  per¬ 
cent  increase  in  the  dollar  value  shipped,  and  no  appreciable  change  in 
the  total  number  of  quality  assurance  manhours  reported.  During  this 
period  a  concerted  effort  was  made  by  the  government  inspection 
agencies,  with  some  success,  to  present  alternate  porposals  to  pur¬ 
chasing  offices  recommending  reductions  in  directed  mandatory  inspec¬ 
tion  effort.  The  bulk  of  this  increase  in  mandatory  inspection  resulted 
from  the  Southeast  Asia  build  up. 

Table  II  shows,  based  upon  April  1966  data,  the  distribution  of 
directed  mandatory  inspection  by  service  and  the  distribution  of  the 
total  dollar  value  inspected  and  shipped  by  service.  These  data  indicate 
an  inconsitency  among  the  purchasing  offices1  policies  for  specifying 
directed  mandatory  inspection.  For  example,  42  percent  of  the  directed 
mandatory  inspection  was  in  support  of  one  Department,  whereas  only 
21  percent  of  the  dollar  value  inspected  and  shipped  was  for  the  same 
Department.  Thirty  percent  of  the  directed  mandatory  inspection  was 
in  support  of  one  agency,,  which  accounted  for  only  10%  of  the  dollar 
vaJ  shipped.  At  the  other  extreme,  another  Department's  procure - 
accounted  for  only  0%  of  directed  mandatory  inspection  and  23% 
of  tne  dollar  value  shipped. 

The  effects  of  a  marked  increase  in  directed  mandatory  inspection 
without  a  commensurate  build-up  in  authorized  manpower  can  only 
result  in  a  decrease  in  activity  in  support  of  other  contracts. 


Recommendations 


It  is  recommended: 

1.  That  OSD  review  and  change  the  present  policies  and  proce¬ 
dures  concerned  with  procurement  quality  assurance  manpower  to  pro¬ 
vide  for  a  capability  to  respond  to  workloads. 

2,  To  obtain  a  reasonable  degree  of  consistency  in  the  expenditure 
of  directed  inspection  effort  between  the  Departments  and  DSA,  that 
OSD  issue  a  policy  statement  setting  forth  the  conditions  for  the 
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Departments  and  DSA  to  specify  directed  inspection  activity.  This 
policy  statement  should  give  cognizance  to  the  relative  importance  of 
the  materials  purchased  by  the  Departments  and  DSA  in  comparison 
to  the  total  military  mission  of  DoD. 

3.  That  DoD  policy  direction  encourage  the  Departments  and  DSA 
to  continue  existing  efforts  or  to  establish  definitive  programs  that 
would  have,  as  its  objective,  the  minimizing  of  directed  mandatory 
inspections.  Full  recognition  must  be  provided  to  the  distinction 
between  individual  or  types  of  products  as  regards  their  relative  impor¬ 
tance  to  the  total  military  mission  of  the  individual  Department. 

4.  That  OSD  thoroughly  explore  the  practicability  of  funding  the 
direct  labor  involved  in  the  CAS  field  quality  assurance  effort  as  an 
administrative  expense  of  procurements. 


Table  I.  Direcced  Mandatory  Inspection  Performed  by 

Government  at  Source 


Dollar  Value  Shipped 

Directed  Mandatory- 
Manhours  (Kand  A  ) 

Mandatory  A  as  %  of 
Total  Reported  M/'H 

DEC 

$1,  142,  100,  000.  00 

26% 

JAN 

1,  182,  700,  000.  00 

241, 416 

27% 

FEB 

1,  175,  800,  000.  00 

235, 836 

28% 

MAR 

1,  451,  800,  000.  00 

302,  622 

31% 

APR 

1,  261,  800,  000.  00 

284,930 

32% 

10%  increase  from 
Dec  65  to  Apr  66 


18%  increase  from 
Jan  to  Apr  66 


23%  increase  from 
Dec  65  to  Apr  66 


Table  II.  Mandatory  Inspection  Vs.  Dollar  Value 
Shipped  -  by  Service 


Distribution  of  Directed 
Mandatory  Inspection  by 
Service _ 


Distribution  of  Total  Dollar 
Value  Inspected  &  Shipped 
_ by  Service _ 


Month  of  April  1966 
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QUALITY  REQUIREMENTS  AND  ACTIVITY  FOR  SMALL 
DOLLAR  PROCUREMENTS 


The  widely  expanded  use  of  the  simplified  purchase  procedures 
especially  in  the  central  procurement  area  over  the  past  few  years  have 
placed  these  procedures  on  a  new  level  of  importance  as  a  procurement 
method. 

The  procedures  have  gained  considerable  favor  as  a  means  of 
reducing  procurement  leadtime  as  well  as  procurement  costs  and 
fostering  competition  in  the  small  dollar  area.  They  have  proven 
particularly  valuable  in  an  environment  where  large  government  stocks 
maintained  at  central  locations  are  being  replaced  by  more  frequent 
buys  of  smaller  quantities  shipped  direct  to  user.  The  advantages 
mentioned  have  been  offset  by  current  ASPR  limitations  which  have 
created  serious  problems  in  the  quality  area. 

ASPR  3-608.  2(b)(l)(ii)  covering  negotiated  procurements  of  less 
than  $2500  prohibits  the  use  of  clauses  covering  subject  matter  of  any 
clause  set  forth  in  ASPR,  other  than  those  set  forth  in  DD  Form  1155 
and  certain  specific  exceptions  approved  by  ASPR.  Several  procuring 
activities  interpret  this  to  preclude  the  use  cf  standard  DoD  quality 
or  inspection  provisions  in  these  procurements  many  of  which  require 
this  form  of  protection. 

In  addition  to  a  requirement  for  the  option  to  use  the  provisions 
mentioned  above,  there  is  an  additional  requirement  for  a  simple 
statement  of  contracto  *  responsibility  for  inspection  of  supplies  and 
maintenance  of  records  of  the  inspections  performed.  This  require¬ 
ment  can  be  expressed  as  follows: 

1.  The  contractor  is  responsible  for  tendering  to  the  government 
only  those  supplies  (which  term  throughout  this  caluse  includes  raw 
materials,  parts  components,  subassemblies  and  end  products  and  the 
identification  and  packaging  thereof)  which  conform  to  all  of  the  require¬ 
ments  outlined  in  the  purchase  order. 

2.  To  fulfill  the  obligat.on  above,  the  contractor  shall  conduct  the 
necessary  inspections  and  tests  of  the  supplies  during  fabrication 
and/or  prior  to  shipment,  as  necessary  to  provide  assurance  that  the 
supplies  meet  the  quality  standards  and  other  technical  requirements 

of  the  purchase  order. 

3.  Records  of  all  inspections  and  tests  performed  by  the  contractor 
on  the  supplies  furnished  shall  be  kept  complete  and  available  (to  the 


1 17 


Government  on  request)  during  the  performance  of  this  purchase  order 
and  shall  be  retained  180  days  after  delivery  of  supplies.  Records 
shall  also  include  a  recording  of  those  checks  made  by  the  contractor 
to  assure  the  accuracy  of  inspection  and  testing  equipment  used  to 
determine  the  acceptability  of  supplies  for  inspection  as  established 
by  DoD  Instruction  4155.  6.  The  lack  of  a  requirement  for  contractor 
maintenance  of  inspection  records  seriously  impairs  the  government 
administering  activity  from  adjusting  the  amount  of  government  inspection 
of  supplies  for  acceptance  purposes  based  on  reliability  of  contractor 
generated  evidence.  This  adjustment  of  government  inspection  activity 
is  set  forth  in  DoD  Instruction  4155.  6. 

The  basic  ASPR  concept  that  "generally  inspection  of  small  purchases 
shall  be  at  destination"  is  not  fully  applicable  to  this  new  area  of  usage 
since  many  of  the  items  cannot  be  inspected  at  destination  and  require 
close  control  during  manufacture.  The  present  concept  of  ASPR  14-106 
on  protection  for  the  government  on  small  purchases  is  primarily  aimed 
at  obtaining  the  replacement  of  correction  of  defective  supplies.  This 
concept  while  appropriate  as  a  base  procurement  philosophy  is  not 
completely  acceptable  under  current  operations  since  delivery  of 
defective  supplies  can  create  serious  problems  of  non-availability  of 
military  equipment.  Repair  lines,  field  maintenance  or  contractor 
field  teams  unable  to  perform  scheduled  work  due  to  defective  supplies 
creates  problems  out  of  all  proportion  to  the  dollar  value  of  the  contract 
or  cost  of  inspection  which  could  have  been  performed  at  source.  Both 
of  these  areas  should  be  thoroughly  studied  by  the  ASPR  Committee  for 
possible  revision  or  clarification.  Caution  should  be  exercised  to  assure 
that  only  those  procurements  meriting  source  inspection  are  so  identified. 

Two  forms  of  Certificate  of  Conformance  (COC)  have  been  in  use 
by  several  procuring  activities  for  some  time  in  addition  to  the  other 
forms  of  inspection  and  acceptance.  One  was  initiated  at  the  option  of 
the  administering  agency  and  used  to  reduce  inspection  workload  in 
facilities  with  good  quality  history.  The  other  is  initiated  by  the  PCO 
and  is  used  in  cases  where  inspection  at  source  cannot  be  performed 
and  inspection  at  destination  is  impractical.  In  this  case  it  is  used  as 
a  form  of  warranty  allowing  time  for  the  government  to  detect  any 
deficiency  in  the  supplies  and  obtain  restitution  from  the  supplier  in 
the  event  deficiencies  are  found.  It  has  been  used  quite  extensively 
when  doing  business  with  distributors,  wholesalers,  etc.  Both  forms 
of  COC  had  certain  utility  and  reduced  the  workload  for  the  administering 
agencies.  This  area  warrants  thorough  study  to  determine  if  COC  is 
the  best  way  to  meet  some  of  the  problems  the  departments  have  been 
trying  to  solve  through  their  use.  If  so,  ASPR  should  recognize  and 
establish  appropriate  guidance. 
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The  number  of  procurements  falling  into  this  category  continues  to 
increase.  One  command  reports  over  86,  000  separate  central  pro¬ 
curement  actions  under  $2500  in  one  12-month  period.  The  Defense 
Contract  Administration  Service  in  a  recent  study  found  in  excess  of 
50,000  contracts  under  $2500  assigned  for  administration.  This  repre¬ 
sented  36%  of  the  contracts  on  hand.  Sixty-six  (66)  percent  of  these 
required  inspection  at  source.  CAS  considers  that  this  situation  absorbs 
government  source  inspection  manpower  that  could  be  better  utilized  on 
supplies  having  higher  relative  importance  to  the  total  military  mission. 

A  service  test  is  currently  under  way  to  test  the  use  of  the  pro¬ 
cedures  on  contracts  up  to  $  10,  000.  If  successful  and  approved  for 
DoD  use,  this  will  further  increase  the  use  of  the  procedures.  While 
there  is  evidence  of  some  abuse  of  the  procedures  and  a  need  for 
policing  by  the  departments,  revision  of  ASPR  could  greatly  improve 
the  situation.  If  the  services  are  to  make  maximum  use  of  the  advan¬ 
tages  inherent  in  the  procedures,  better  protection  and  additional 
guidance  must  be  provided. 


Recommendation 

It  is  recommended  that  an  ASPR  case  be  established  to  investigate 
the  simplified  purchase  procedures  with  the  objective  of  allowing  maxi¬ 
mum  use  of  the  advantages  inherent  in  the  procedures  while  maintaining 
necessary  government  quality  protection. 
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PROCUREMENT  QUALITY  ASSURANCE  INTERFACE  PROBLEMS 


At  the  ASD  management  level  and  at  the  grass  root  operating  level 
there  are  actions  underway  in  the  quality  and  reliability  areas  that 
indicate  that  these  two  elements  of  DoD  are  not  compatible.  This  is 
typified  by  such  programs  as  systems  management,  system  effective¬ 
ness,  total  procurement  package  concepts  and  others  which  highlight 
special  attention  to  commodities  and  management  concepts  that  are  in 
the  limelight  at  a  particular  point  in  time.  These  concepts  do  not 
recognize  the  needs  of  the  long  range  DoD  quality  and  reliability 
requirements. 

Existing  DoD  policies  and  concepts  fail  to  recognize  the  importance 
of  a  total  quality  and  reliability  program.  More  basically  they  fail  to 
recognize  that  there  are  several  methods  or  approaches  used  in 
development  of  a  weapon  system. 

Essentially  there  are  two  general  approaches  followed  in  the 
development  of  defense  material: 

1.  Development  is  by  the  government  at  one  of  its  Engineering 
Design  Centers. 

2.  Development  and  production  is  by  American  industry. 

There  are  advantages  and  disadvantages  in  both  of  these  basic  methods. 
Regardless  of  the  benefits,  these  approaches  are  considered  to  be  the 
two  procedures  available  to  the  Department  of  Defense  in  development 
of  its  complex  weapons  systems. 

Those  items  that  are  developed  in-house,  so  to  speak  (by  the 
Government  engineering  centers),  are  developed  realizing  that  mass 
production  will  be  on  a  competitive  procurement  basis  and  definitive 
drawings,  specifications  and  supporting  data  will  be  prepared  as  the 
item  is  developed.  Items  that  are  developed  by  American  industry 
normally  go  to  that  industry  for  at  least  the  first  production  contract 
for  the  item.  Thus  the  technical  documentation  for  these  items  is  not 
as  detailed  and  complete  as  those  items  designed  in-house  by  the 
Government. 

Existing  DoD  policies  fail  to  recognize  the  total  quality  and 
reliability  requirements  and  the  variation  in  the  technical  documentation 
for  items  developed  in-house  as  opposed  to  those  developed  by  industry. 
Essential  quality  considerations  in  the  development  phase  are  either 
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missing  or  have  not  been  coordinated  with  the  production  quality 
assurance  personnel.  Major  decisions  affecting  quality,  made  during 
production,  are  often  not  available  to  the  maintenance  and  operating 
function.  Further,  this  fragmentation  has  expanded  at  the  departmental 
level  and  has  resulted  in  a  wide  variation  in  the  methods  utilized  in 
determining  product  quality.  There  are  several  DoD  Handbooks  which 
provide  uniform  guidance  in  some  areas  such  as;  quality  program 
evaluation,  statistical  sampling,  calibration  requirements,  etc.  Never¬ 
theless,  these  handbooks  are  not  sufficient  for  implementing  a  uniform 
system  for  determining  product  quality. 

The  establishment  of  DCAS  highlighted  the  problem  of  ''Procedures'1 
and  the  necessity  for  a  standard  set  of  procedures  to  be  applied  to  the 
bulk  of  DoD  procurements  and  material.  However,  the  wide  variation 
in  the  methods  used  by  the  military  technical  activities  in  specifying 
quality  and  reliability  requirements  is  having  an  adverse  effect  on  the 
effective  implementation  of  these  procedures.  Further,  since  DCAS 
procedures  were  never  circulated  for  official  service  coordination  and 
were  never  service  tested,  they  have  not  been  totally  accepted  by  the 
military  departments  for  assuring  the  quality  of  some  of  their  items. 

ASPR  1-108  now  dictates  that  no  departmental  procedures  are  to 
be  published  to  supplement  the  ASPR.  Thus  a  single  set  of  procurement 
quality  assurance  procedures  applicable  to  most  of  the  DoD  procure¬ 
ments  must  be  developed.  The  absence  of  a  total  quality  and  reliability 
program  which  recognizes  the  different  methods  of  developing  defense 
weapons  systems  makes  this  a  most  difficult  task. 

This  panel  considers  it  essential  that  aggressive  action  be  taken 
to  establish  a  single  set  of  procurement  quality  assurance  procedures 
which  can  be  effectively  and  efficiently  applied  to  the  bulk  of  DoD  pro¬ 
curements  and  materiel.  This  single  set  of  procedures  should  be 
published  as  a  supplement  to  the  ASPR. 


Recommendations 


It  is  recommended; 

1.  That  the  Deputy  Secretary  of  Defense  establish  a  task  group;  to 
define  and  develop  a  total  quality  and  reliability  program  covering  the 
entire  product  lift  cycle,  or  to  develop  a  system  that  will  assure  that 
adequate  and  just  considerations  are  given  to  quality  and  reliability- 
requirements  throughout  the  entire  product  life  cycle. 
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2.  That  DoD  recognize  in  its  policies  and  procedures  that  there 
are  two  methods  of  developing  defense  material  (weapon  systems)  and 
that  there  is  a  distinct  difference  in  the  type  of  technical  documentation 
that  results  from  these  two  methods  of  development. 

3.  That  ASD  (I&L)  assure  that  positive  and  timely  actions  are 
taken  to  develop  a  uniform  set  of  quality  assurance  procedures  that 
can  be  effectively  and  efficiently  applied  to  the  bulk  of  DoD  procure¬ 
ments  and  that  will  satisfy  the  requirements  and  intent  of  ASPR  1-108. 


REPORT  OF  PANEL  4 


TITLE:  Quality  Assurance  in  Storage  Operations 
OBJECTIVE: 

To  assure  the  quality  and  reliability  of  materiel  being  issued 
for  use  from  storage  operations. 

TOPICS  DISCUSSED: 

1.  Quality  of  Materiel  Upon  Receipt  by  Storage  Activities 

2.  Quality  Deterioration  of  Material  in  Storage 

3.  Minority  Positions 


123 


PARTICIPANTS 


Chairman 


Panelists 


Recorder 


Monitor 


Mr.  Warren  S.  Koontz, 

Naval  Ordnance  Systems  Command 

Mr.  B.  A.  Matulaitis,  HQS,  Air  Force 

Mr.  Charles  Ramsey, 

Middletown  Air  Materiel  Area 

Mr.  L.  Weisse  *ilw  w  rger, 

FAA  Depot,  Oklahoma  City 

Mr.  Robert  H.  Ford, 

Naval  Facilities  Engineering  Command 

Colonel  T.  C.  McNeal,  USAF,  DCASR,  Cleveland 

Mr.  Elmer  Ward,  Defense  Depot,  Ogden,  Utah 

Mr.  Don  Foy,  Letterkenny  Army  Depot 

Mr.  David  H.  Magathan,  HQS,  Army 

Mr.  Roger  F.  Carroll,  Jr.  ,  FSS,  GSA 

Mr.  Lawrence  E.  Grey,  Jr.  , 

Naval  Ammo  Depot,  Oahu 

Mr.  Gordon  J.  Keefe,  HQS,  DSA 


124 


QUALITY  OF  MATERIAL  UPON  RECEIPT 
BY  STORAGE  ACTIVITIES 


1.  Definition  of  Storageability 

There  is  considerable  evidence  of  uneven  practices  and 
incompatibilities  between  the  various  services  concerning  thousands  of 
line  items  assigned  (often  arbitrarily)  limited  shelf  life  or  service  life. 
This  is  particularly  evident  in  the  area  of  elastomeric  materials,  e.  g.  , 
rubber  goods,  o-rings,  gaskets,  etc. ,  '"here  their  shelf -life  often 
arbitrarily  governs  the  overhaul  or  rework  intervals  of  major  items. 
Another  good  example  is  acrylic  aircraft  paints  where  shelf  life  set  by 
one  agency  is  often  twice  that  of  another,  even  though  it  is  an  item 
common  to  all  services.  It  is  believed  that  the  formalization  and 
standardization  of  "storageability"  requirements  would  provide  signifi¬ 
cant  improvements  to  the  quality  of  material  in  storage  and  further 
would  have  a  salutary  effect  upon  the  maintenance  activity  concerned. 

There  are  many  known  instances  where,  by  hindsight,  it  is  obvious 
that  if  a  few  extra  pennies  had  been  spent  to  change  or  add  a  metal 
plating,  or  to  make  a  minor  change  in  packaging,  or  some  other  usually 
insignificant  change  there  would  have  been  a  drastic  increase  in  shelf - 
life  reliability.  It  is  felt  that  firm  definitions  and  requirements  for 
"storageability"  should  be  considered  in  the  design  phase  along  with 
reliability,  maintainability  and  value  engineering  in  order  that  storage 
life  of  the  end  item  be  based  on  reasoned  dec  sions  rather  than  made 
arbitrarily. 

It  is  well  understood  by  the  Panel,  that  from  a  technical  standpoint, 
"storageability"  is  nothing  new.  Every  designer,  packaging  engineer, 
etc.  has  always  been  involved.  The  same  can  be  said  for  reliability, 
maintainability  and  value  engineering.  It  is  believed  that  the  same 
reasoning  that  was  used  to  define  and  separate  these  latter  disciplines 
is  equally  applicable  to  "storageabilit  I;  By  giving  "storageability" 
requirements  proper  definition  and  adequate  visibility  early  enough  in 
the  design  phase,  and  incorporating  them  in  the  production  technical 
data  package,  considerable  improvement  in  storage  reliability  with 
resultant  cost  savings  should  be  effected. 


Recommendation 

It  is  recommended  that  OSD  publish  a  DoD  Directive  requiring 
Services  to  spell  out  requirements  for  "storageability"  in  all  major 
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procurement  and  in  all  procurement  of  material  known  to  deteriorate 
seriously  with  time.  It  is  recommended  that  the  Defense  Supply  Agency 
be  designated  to  develop  this  directive. 

2.  Inspection  and  Acceptance  Requirements 

Recent  and  continuing  studies  by  Navy,  Air  Force,  and  the 
Aerospace  Industries  Association,  among  others,  have  come  up  with 
remarkably  consistent  findings  on  the  quality  of  material  being  received 
by  the  Services  particularly  in  the  areas  of  spare  parts  procurement. 

The  percentage  of  material  received  that  fails  to  meet  specifications 
ranges  between  25  and  35  percent.  About  half  of  this  percentage  is 
unusable  "as  is"  by  the  Armed  Forces.  It  must  be  borne  in  mind  that 
"spare  parts"  usually  cover  the  great  majority  of  all  components  making 
up  a  weapon  system  or  other  military  equipment,  e.  g.  ,  every  part  of  a 
military  aircraft  is  given  spare  part  support  except  the  airframe  (less 
wings).  In  the  instance  of  the  Navy  investigators,  most  data  on  defective 
material  were  derived  by  exhaustive  testing  of  the  components  involved 
by  engineers  and  technicians  in  Navy  laboratories  and  therefore  the 
findings  have  complete  technical  documentation  backup.  Percentages 
of  this  material  now  ranging  up  to  50%  are  shipped  directly  to  the  storage 
or  user  activities  under  contractual  arrangements  specifying  inspection 
and  acceptance  at  destination.  This  material  has  not  been  inspected  at 
source  and  the  onus  therefore  is  placed  upon  the  storage  activity  to 
inspect  (or  test)  as  necessary.  In  most  instances,  the  storage  activity 
has  not  been  provided  with  funds,  personnel  or  necessary  test  and 
inspection  facilities.  He  can  provide  normally  only  an  "eyeball  inspec¬ 
tion"  of  packaging  and  marking. 

The  studies  indicate  that  excessive  defective  material  is  being 
received  from  both  major  industry  and  small  business  sources.  The 
concept  of  holding  prime  contractors  responsible  for  the  quality  of 
their  vendors'  material  is  more  often  than  not  quickly  deteriorated  by 
"break-out  buys",  "fast-pay"  contract  procedures,  particularly  the 
areas  of  spare  parts  and  follow-on  procurement.  It  is  common  know¬ 
ledge  that  former  known  high-quality  producers  have  been  consistently 
underbid  due  to  their  unwillingness  to  sacrifice  quality.  It  is  interesting 
to  note  that  many  of  these  same  contractors  are  known  quality  producers 
of  important  commercial  items  and  they  are  still  able  to  compete 
successfully  for  their  business.  The  findings  of  the  AIA  Quality 
Assurance  Panel  (Mr.  J.  Motolina,  Jr)  indicate  that  much  of  industry 
top  management  is  not  personally  aware  of,  or  does  not  personally 
understand  these  problems. 

It  is  considered  that  the  DoD  must  recognize  this  immediate  situation 
because  it  impacts  severely  upon  quality  of  material  in  storage  and  in 
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the  hands  of  the  combat  forces.  The  only  known  means  to  correct  the 
problem  are  by  providing  additional  resources  to  the  storage  activities 
or  by  requiring  valid  source  acceptance  inspection.  It  is  again  empha¬ 
sized  that  DoD  materials  in  storage  go  directly  to  the  aircraft  squadrons. 
Armies,  and  Fleets  for  combat  use  and  thus  action  at  this  level  repre¬ 
sents  the  last  chanCe  for  the  user. 


Recommendation 

It  is  recommended  that  DoD  take  necessary  action  to  eliminate  or 
reduce  drastically  the  awarding  of  "acceptance  at  destination"  contracts, 
including  as  a  minimum  all  items,  which  if  defective,  would  cause 
malfunctions  or  casualties  of  military  weapons  and  combat  equipments, 
or  amend  DoD  Directive  4155.  1 1  to  require  all  storage  activities  to 
inspect  and/or  test  material  received  under  "acceptance  at  destination" 
contracts  at  the  time  of  receipt  from  vendors/contractors,  and  provide 
the  resources  to  do  so. 

* 

3.  Organization* 

The  intent  of  Para  V  B,  the  Responsibility  Section,  of  the  DoD 
Directive  4155.  11  is  not  clear  as  to  the  organizational  requirements 
for  management  of  Quality  Assurance  functions.  Para  V  B  states: 

"The  policy  direction  and  responsibility  for  implementing  DoD  problems 
on  Quality  Assurance  and  Reliability  Assurance  of  the  material  pro¬ 
cured  or  produced,  maintained,  and  stored  shall  be  centralized  within 
offices  specified  by  the  Secretaries  of  the  Military  Departments  and 
the  Director,  Defense  Supply  Agency.  The  Quality  and  Reliability 
Assurance  Policy  function  should  be  managed  independently  of  other 
functions  (e.  g.  ,  procurement,  production,  maintenance  and  storage), 
but  within  approved  manpower  authorizations.  This  requirement  does 
not  preclude  the  grouping  of  various  related  functions  within  a  large 
organization.  "  Experience  of  the  various  military  services  has  shown 
that  the  last  sentence  has  made  it  possible  to  interpret  this  paragraph 
in  many  ways.  The  response  to  the  directive  has  varied  from  establish¬ 
ment  of  an  independent  quality  -\ssurance  agency  to  no  action  at  all 
since  the  functions  were  already  grouped  within  a  large  organization. 
Apparently  the  intent  of  the  directive  was  to  clarify  the  responsibility 
for  the  independent  management  of  Quality  and  Reliability  Assurance 
functions.  If  so,  the  directive  has  not  succeeded. 


^ee  minority  position  on  <uhis  subject  on  page  132. 


Recommendation 
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It. is  recommended  that  OSD  (I&L)  amend  DoD  Directive  4155.  11 
(Para  V.  B.  )  to  delete  authorization  to  combine  quality  assurance 
functions  with  other  storage  operations,  if  the  intent  of  the  directive  is 
to  require  independent  quality  and  reliability  management.  (See  minority 
positions  on  this  recommendation  at  end  of  report.) 

4.  Identification  of  Functions 

The  requirement  for  identification  of  the  functions  in  DoD 
Directive  4155.  11  that  are  to  be  performed  by  the  depot  organization 
responsible  for  execution  of  the  storage  quality  and  reliability  program 
is  not  specific.  The  following  is  a  list  of  functions  that  should  be 
specified:  ° 

1.  Perform  procurement  inspection/acceptance 

2.  Identify/ classify  returned  materiel 

3.  Inspect  stock  transfers 

4.  Maintain  quality  history  data 

5.  Schedule/perforrn  cyclic  inspection 

6.  Re -identify/ reclassify  materiel 

7.  Request  disposition  from  commodity  manager 

8.  Perform  stock  investigation 

9.  Perform  issue  inspection 

10.  Record/report  inspection  results  (data  feedback) 

11.  Inspect  for  modification  work  order  compliance 

12.  Maintain  technical  data  files 

13.  Conduct  packaging  inspection 

14.  Inspection/quality  assurance  planning 

15.  Perform  evaluation  and  verification 

16.  Provide  technical  advice  and  assistance 

These  functions  have  been  divided  into  three  segments  as  indicated. 
The  nine  functions  in  the  top  segment  are  those  which  should  be  per¬ 
formed  by  a  single  organizational  element.  Items  10  through  14  are 
activities  common  to  all  of  the  first  nine  functions.  Items  15  and  16 
are  assurance  actions  that  are  common  to  all  of  the  preceding  functions. 
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Recommendation 

It  is  rec  mmended  that  DoD  Directive  4155.  11  be  amended  to 
specify  those  quality  control  and  reliability  functions  to  be  performed 
by  a  storage  quality  control  organization. 

5.  Identification  of  Material 

Storage  activities  report  that  considerabia  quantities  of  material 
are  received  with  contractor  part  number  or  sub -contractor  part  number 
markings  only.  Identification  of  these  materials  creates  a  severe 
quality  problem  and  is  often  responsible  for  the  improper  issue  of  the 
material.  DSA  has  initiated  action  to  require  proper  identification  as 
part  of  the  contract  requirements. 


Recommendation 

It  is  recommended  that  material  being  procured  for  storage 
activities  (or  for  the  operating  forces)  be  marked  for  identification  by 
Federal  Stock  Number,  where  such  numbers  have  been  assigned. 
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QUALITY  DETERIORATION  OF  MATERIAL  IN  STORAGE 


1.  Serviceability  Standards 

All  Services  report  deficiencies  in  the  area  o i  Serviceability 
Standards.  In  the  fields  of  explosive  items  and  other  hazardous 
materials,  Army,  Navy  and  Air  Force  nave  apparently  adequate  pro¬ 
cedures  and  standards.  Particularly  in  the  case  of  Navy,  other 
materials  are  sparsely  covered. 

Panel  4  concurred  that  to  provide  adequate  coverage  of  all  materials 
involved  would  require  technical  capabilities  (duplication)  not  usually 
available  to  the  storage  activities  to  prepare  such  criteria.  Also  many 
items  are  used  commonly  between  the  services. 

It  was  therefore  concluded  that  the  only  possible  sources  of  these 
Serviceability  Standards  would  be  available  from  the  overall  resources 
of  the  Commodity  Managers  for  the  particular  items  involved,  and  that 
where  commonality  exists  that  one  agency  should  be  designated  to  pre¬ 
pare  the  standards. 


Recommendation 

It  is  recommended  that  material  managers  such  as  Commodity 
Managers,  Project  Managers,  Project  Officers,  System  Program 
Director,  etc.  ,  be  required  to  provide  storage  activities  with  Storage 
Serviceability  Standards  for  material  receipt,  storage,  and  issue  in 
storage  operations.  These  standards  must  define  the  minimum  level 
to  which  deterioration  can  progress  without  impairment  of  service¬ 
ability  and  user  satisfaction.  It  is  further  recommended  that  the  Army 
be  designated  to  prepare  a  DoD  directive  to  this  effect. 

2.  Utilization  of  Evaluation  Facilities  and  Data 

A  considerable  body  of  quality  and  reliability  data  on  material 
in  storage  has  been  generated  by  test,  evaluation,  and  maintenance 
functions.  In  some  specific  cases  data  have  been  fed  back  to  design 
agencies,  but  for  the  most  part  this  information  is  not  available  in 
usable  form  or  if  it  is  available  it  is  not  used  by  design  activities.  There 
have  been  efforts  to  systematize  portions  of  this  body  of  data,  partic¬ 
ularly  in  the  area  of  materials  deterioration,  but  these  efforts  have  not 
been  supported  consistently.  For  instance,  the  w'ork  done  by  the 
National  Science  Foundation  Center  for  the  Prevention  of  Deterioration 
under  DoD  sponsorship  has  been  stopped.  Perhaps  this  effort  was  not 
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responsive  to  the  needs  cf  design  agencies,  or  perhaps  the  existence  of 
the  center  was  not  sufficiently  publicized  so  that  design  engineers 
would  utilize  the  information  available.  Similar  efforts  to  systematize 
storage  quality  data  have  been  made  by  various  service  activities,  but 
these  have  usually  been  specialized  in  nature  and  accessible  only  to  a 
limited  group.  There  is  a  distinct  vacuum  in  the  area  of  technical 
data  exchange  regarding  known  deterioration  rates  and  factors  of 
common  items  in  storage. 

Similarly,  the  services  all  have  test  and  evaluation  facilities,  some 
highly  specialized  and  some  general  in  nature.  One  example  of  highly 
advanced  capability  in  this  area,  is  the  Navy's  Quality  Evaluation 
Laboratory  complex.  This  group  of  activities  represents  the  most 
well  equipped  and  staffed  group  in  the  CONUS  and  Hawaii  in  the  fields 
of  both  Destructive  and  Non-Destructive  testing.  Some  cross -servicing 
is  provided  in  these  facilities  but  normally  the  established  workload 
is  such  that  utilization  by  other  services  cannot  be  accommodated. 
Arrangements  to  increase  workload  in  these  facilities  will  require 
study  of  potential  capacity  and  requirements  for  additional  personnel 
and  funding  support. 


Recommendation 

It  is  recommended  that  utilization  of  storage  and  user  test  and 
evaluation  data  be  made  mandatory  and  that  necessary  arrangements 
be  made  at  the  DoD  level  to  utilize  skills  and  facilities  unique  to  any 
one  service  by  all  services  in  the  quality  evaluation  of  stored  and 
service-held  material, 

3,  Effect  of  Periodic  Testing  on  Quality 

Large  scale  tests  in  Navy  on  complex  guided  missiles,  e.  g.  , 
TARTAR,  show  that  missiles  which  have  not  been  given  the  usual 
periodic  storage  and  shipboard  tests  perform  about  10%  higher  in  actual 
flight  than  missiles  which  have  been  repetitively  tested.  The  TERRIER 
and  SPARROW  III  results  appear  to  be  generating  the  same  conclusions. 
The  result  in  cutback  of  tests  of  these  type  materials  in  storage  pro¬ 
vides  a  drastic  cutback  in  cost  of  missile  maintenance. 


Recommendation 

It  is  recommended  that  OSD  assign  the  Air  Force  and  Navy  to 
investigate  the  results  of  routine  periodic  storage  testing  of  complex 
items  (particularly  electronic)  on  a  selective  basis,  and  weigh  reli¬ 
ability  degradation  from  these  tests  versus  quality  assurance  gains. 
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MINORITY  POSITIONS 


Organization 


The  AF  representative  (Hq  USAF,  AFSSSDC)  does  not  agree  that 
Par  V  B,  DoD  Directive  4155.  11  presents  a  problem.  Policy  direction 
and  responsibility  as  outlined  therein  has  been  defined  within  the  Air 
Force.  Further  clarification  by  OASD  (I&L)  is  not  required.  The 
spirit  and  intent  of  the  DoD  Directive  4155.  11  will  be  carried  out  under 
the  present  Air  Force  organizational  structure. 

(Signed)  Ben  A.  Matulaitis 

Hq  USAF  (AFSSSDC) 

Member,  Panel  4 

The  discussion,  solution  and  action  relative  to  the  subject  problem 
presented  by  the  panel  with  exception  of  the  DAF  representative,  indi¬ 
cates  grave  problems  exist  because  DoD  Directive  4155.  11  allows 
flexibility  by  providing  that  quality  assurance  policy  functions  may  be 
grouped  with  others  in  a  larger  organization.  Although  these  problems 
and  their  attendant  impacts  are  not  stated,  it  is  implied  that  quality 
assurance  functions  and  programs  of  the  services  are  suffering  because 
of  the  flexibility  currently  provided  in  the  Directive.  Apparently  con¬ 
curring  panel  members  feel  it  necessary  that  quality  assurance  must 
have  an  organization  separate  and  apart  from  all  others  before  it  can 
be  effective. 

The  writer  believes  that  the  policy  guidance  provided  by  DoD 
Directive  4155.  11  is  provocative,  indicative  yet  flexible  as  to  its 
intent.  It  is  not  believed  it  was  DoD's  intent  .r  unilaterally  dictate  to 
the  service  secretaries  or  Director  DSA  as  to  how  they  are  to  organize 
within  their  headquarters  staffs,  operations  agencies  and  depots  to 
accomplish  the  quality  assurance  functions  assigned.  It  is  also  not 
believed  that  DoD  intended  that  t  he  services  and  DSA  reorganize  to 
create  new  independent  staff  and  operational  elements  to  accomplish 
the  quality  assurance  and  reliability  functions. 

The  writer  therefore  does  not  agree  that  Paragraph  V  B,  DoD 
Directive  4155.  11  presents  a  significant  problem  particularly  with 
respect  to  the  successful  accomplishment  of  the  storage  quality  assu¬ 
rance  and  reliability  program  at  depot  level  or  that  the  Directive 
requires  clarification  as  to  its  intent. 

(Signed)  David  H.  Magathan 

Representative,  ODCSLOG 
Hq,  Department  of  the  Army 


REPORT  OF  PANEL  5 


TITLE:  Quality  Assurance  in  Maintenance  Operations 
OBJECTIVE: 

To  assure  that  material  reconditioned,  maintained  and 
modified  for  the  Military  Services  meets  requisite  quality. 

TOPICS  DISCUSSED: 

1.  Consistent  Quality  Assurance  at  all  Echelons  of 
Maintenance 

2.  Continuity  of  Consecutive  Modifications 

3.  Inspection  Cognizance  for  Contract  Maintenance 

4.  Reliability  Management  relating  to  Maintenance 

5.  Relationship  between  Reliability  Analysis  and  Preventive 
Maintenance 

6.  Reconditioning  Standards 

7.  Quality  Standards  for  MAS  and  Grant  Aid  Materiel 

8.  Improved  Reconditioning/Serviceability  Standards  for 
Attribute  Characteristics 

9.  Use  of  Contractor  Rejected  Material  in  Maintenance 
Activities 

10.  Impact  of  Contract  Waivers  on  Maintenance 

11.  Quality  Control/Inspection  Skills  Gap 

12.  Reducing  Maintenance  Workload  Through  More  Complete 
"Use"  Instruction 

13.  Maintenance  Technical  Data 

14.  Deficiency  Data  Feedback 
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CONSISTENT  QUALITY  ASSURANCE  AT  ALL  ECHELONS 

OF  MAINTENANCE 


Ideally,  all  maintenance  tasks  (regardless  of  level)  should  be  accom¬ 
plished  without  flaw,  thereby  eliminating  the  necessity  for  inspection. 

In  reality  such  utopian  goals  are  seldom  achieved,  since  humans  are 
subject  to  errors  of  both  action  and  judgment.  To  make  matters  worse, 
managers  are  often  confronted  with  the  situation  where  repairmen  are 
guided  with  less  than  adequate  technical  instruction,  and  frequently  lack 
the  motivation  to  produce  quality  work  on  a  sustained  basis.  The  lack 
of  uniform  inspection  procedures  for  repair  work,  coupled  with  the  prob¬ 
lem  of  selecting  or  identifying  suitable  personnel  specialty  codes  for 
continued  assignment  to  quality  control  duty,  has  been  noted  as  a  signif¬ 
icant  shortcoming,  especially  at  the  organizational  maintenance  level. 
Additionally,  the  lack  of  a  method  by  which  management  can  measure 
the  effectiveness  of  local  quality  control  programs  is  noted  as  a  predom¬ 
inant  weakness  in  achieving  improvement  at  all  level0. 

Although  inadequate  inspection  procedures  and  the  lack  of  identity 
for  quality  control  specialty  codes  are  rather  widely  recognized  problems, 
most  Brambles  of  the  Services  have  directives  and  safeguards  designed  to: 

1.  Establish  adequate  standards  of  quality. 

2.  Stipulate  the  in-process  point  or  time  when  inspection  must 
occur. 

3.  Allocate  adequately  trained  hardware  specialists  to  a  quality 
control  function. 

4.  Delegate  authority  and  provide  the  resource  to  incure  correction 
of  defects  found  during  the  inspection  process. 

While  there  are  often  serious  deficiencies  in  the  quality  control  functions 
operating  within  this  framework,  in  the  main,  all  the  elements  of  a 
rudimentary  program  exist. 

Since  the  maintenance  quality  control  function  is  deficient  as  it 
presently  exists,  and  varies  widely  even  within  a  single  Military  Service, 
it  is  evident  that  a  management  plan  and  associated  funding  budget  are 
needed  to  standardize  the  program  and  provide  some  deg*£e  of  opera¬ 
tional  consistency.  There  is  an  attendant  need  for  manageme^v  to  be  able 
to  evaluate  the  effectiveness  of  quality  programs  in  order  to  r;ng  about 
the  required  improvements.  Further,  the  management  plan  ..ust  provide 
for  a  clear,  well-defined  channel  of  communication,  fundinf  equirements 
and  other  associated  quality  assurance  problems  throughc  ad  echelons 
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of  maintenanace  within  and  between  Services.  The  magnitude  of  the 
maintenance  program,  both  in  terms  of  manpower  and  dollars,  dictates 
that  this  problem  receive  immediate  DoD  attention. 


Recommendations 


It  is  recommended: 

1.  That  a  DoD  directive  patterned  after  Nav  Ships  Instruction 
5450. 148  -  Standard  Naval  Shipyard  Regulations  be  prepared  which  will 
strengthen  quality  assurance  and  provide  total  program  orientation 

for  all  levels  of  maintenance.  The  directive  must  establish  the  concept 
of  operation  and  delineate  organizational  lines  of  quality  assurance 
communication  and  responsibilities  (from  top  to  bottom)  for  the  mainte¬ 
nance  programs  for  each  Service  and  lead  to  improved  consistency  at 
all  echelons. 

2.  That  DoD  initiate  a  Tri -Service  Project  to  test  the  use  of 
Mil-Q-9858A  (Quality  Program  Requirements)  con  ept  in  depot  overhaul 
facilities.  This  test  to  be  confined  to  one  depot  per  Service. 

3.  A  DoD  task  group  be  established  to  develop  a  handbook  for 
general  guidance  in  performing  maintenance  quality  control.  This 
handbook  would  be  useful  to  organizational  level  maintenance  managers 
in  developing  management  plans  that  are  generally  consistent  with  the 
depot/contract  program. 

4.  That  DoD  initiate  action  to  identify  quality  control  specialists, 
both  civilian  and  military,  with  a  functional  code  in  order  to  provide 
for  continuity  of  duty  assignment. 

5.  That  DoD  require  the  Services  to  establish  within  their  budget 
an  identified  funding  structure  of  appropriate  breakout  to  cover  all 
aspects  of  the  quality  control  program  as  related  to  maintenance  activi¬ 
ties  at  depot  level  and  above. 
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CONTINUITY  Oi  CONSECUTIVE  MODIFICATIONS 


Weapon  systems  design  modifications  are  often  carried  out  without 
appropriate  recognition  of  pertinent  preceding  actions,  or  those  that 
may  follow.  Frequently  modifications  are  completed,  only  to  receive 
another  modification  which  requires  disassembly  and  reassembly.  It 
is  not  uncommon  for  modifications  to  reach  equipment  out  of  sequence, 
with  the  effect  that  modifications  are  sometimes  undertaken  on  equip¬ 
ment  not  to  the  latest  "configuration."  This  practice  increases  the 
chance  of  contaminating  the  unit  as  well  as  contributing  to  the  likeli¬ 
hood  of  serious  maintenance  errors.  Thus,  the  improved  reliability 
inherent  in  the  modification  may  be  negated,  with  an  additional  waste 
of  maintenance  dollars. 

The  need  for  a  centralized  system  of  planned,  orderly  control  of 
the  modification  process  i  ;  o  ,  dent.  This  control  should  be  time  phased, 
yet  flexible  enough  to  handle  situations  of  urgency  that  may  affect  safety. 
Also,  scheduled  modification  actions  relate  directly  to  scheduled  mainte¬ 
nance  actions. 

Analysis  of  approximately  1000  check-in  discrepancies  of  a  ship¬ 
board  weapon  installation  revealed  over  75%  of  the  total  were  the 
combined  effects  of  poor  installation  and  maintenance  practices.  Not¬ 
withstanding  usual  design  and  production  deficiencies,  such  statistics 
indicate  additional  responsibility  in  areas  relating  to  controlled  handling 
of  modifications  and  maintenance. 

It  is  frequently  very  difficult  to  isolate  the  inter -faces  of  pure 
design  complicity  vs  pure  maintenance  when  dissecting  mass  deficiency 
data.  Therefore,  a  host  of  "make-do"  modifications  often  evolve  which 
provide  an  interim  fix  to  locally  identified  problems.  The  hazard  of 
this  modus  operandi  is  that  later  modifications  sometimes  negate  the 
result  of  an  earlier  fix,  or  in  correcting  deficiencies  associated  with 
one  functional  area  (e.  g.  ,  design)  result  in  the  occurrence  of  a  new 
problem  appearing  in  another  function  area  (e.  g.  ,  quality  control). 

The  system  must  recognize  the  need  for  introducing  conceptual  and 
developmental  inputs  into  the  system  life  cycle  in  an  effort  to  minimize 
the  amount  of  "continuing  engineering"  performed  on  the  item. 
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Recommendation 


It  is  recommended  that  a  more  effective  centralized  Service  control 
and  review  of  maintenance  modification  publications  be  established  to 
insure  timely,  consecutive  scheduled  incorporation  of  modifications. 
This  recommendation  should  also  be  incorporated  in  the  DoD  configura¬ 
tion  management  program. 
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INS  I-  ACTION  COGNIZANCE  FOR  CONTRACT 
MAINTENANCE1 


To  insure  consistent  quality  of  contract  and  depot  maintenance, 
there  is  a  need  to  re-examine  those  specialized  contract  maintenance 
facilities  to  ascertain  which  should  be  reassigned  to  the  contracting 
Service. 

Effective  maintenance  management  dici/ites  that  the  System  Support 
manager  must  have  more  direct  control  and  access  for  the  exercise  of 
technical  direction  over  all  specialized  maintenance  and  repair  activities 
engaged  in  the  maintenance,  overhaul  repair  and  modification  of  the 
weapons  for  which  he  is  charged.  This  especially  includes  quality  control 
and  the  planning  functions  related  thereto. 

As  an  extension  of  the  organic  depot  level  Service  activity,  contract 
maintenance  is  equally  a  part  of  the  SSM's  responsibility.  Administration 
of  this  type  contract  by  an  agency  other  than  the  initiating  Service  poses 
special  problems  in  the  input  of  reparables,  interpretation  of  work 
specifications,  supply  support,  and  quality  control  (from  a  manpower 
standpoint,  the  latter  function  represents  40-45%  of  the  in-plant  Govern¬ 
ment  workload).  Split  responsibilities  in  these  areas  have  resulted  in 
communications/control  problems  -  primarily  as  relates  to  interpretation 
of  requirements  -  that  has  effected  quality  control  performance. 

The  principal  problems  pertaining  to  quality  control  at  contract 
maintenance  activities  is  related  to  the  extremely  difficult  problem 
of  writing  rework  specifications  for  a  product  which  is  produced  from 
a  RAW  material  input  (reparable  item)  which  is  non-uniform  in;  age, 
source,  configuration,  expended  service  life  and  as  received  condition. 

The  nature  of  the  work  performed  to  produce  the  product  is  also 
highly  variable;  some  items  require  major  rework  and  modification, 
others  require  only  minor  adjustment  and  repair.  Under  these  conditions 
the  acceptable  level  of  quality  is  not  fixed.  Contract  administration 
(depending  upon  the  type  of  contract)  ranges  from  trying  to  prevent  the 
contractor  from  Gold  Plating,  (CPFF  and  Job  Order  Contracts)  to  trying 
to  insure  an  acceptable  level  of  performance  (fixed  priced  contracts). 

In  a  situation  with  this  iarge  number  of  independent  variables  it  is 
essential  to  have  "user  oriented"  quality  control  personnel  who  are 
well  versed  in  technical  specifics  and  in  close  contact  with  the  using 
Services  technical  authorities. 


1  See  minority  position  on  this  subject  on  page  141. 
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A  related  problem  is  the  necessity  for  performing  analytical 
engineering  examinatior  of  failed  items  received  for  rework.  These 
examinations  may  be  for  the  purposes  of  revising  engineering  or  mainte¬ 
nance  deficiencies,  investigating  accidents,  developing  inputs  to  improve 
schedules  maintenance  component  life  limitations  or  rework  specifica¬ 
tions  or  to  verify  or  determine  specific  problems  defined  in  mechanized 
data  collection  systems.  These  analyses  are  closely  related  to  internal 
functions  of  the  using  Services  md  should  be  performed  by  personnel 
who  also  arc  well -versed  in  the  technical  specific  and  technical  manage¬ 
ment  practices  of  the  using  Service. 

The  Defense  Contract  Administration  Services  (DCAS)  organization 
was  established  as  the  central  DoD  contract  administration  Service 
organization,  and  although  it  was  originally  planned  that  DCAS  would  be 
responsible  for  contract  management  of  all  plants  performing  under 
contract  to  the  Military  Services,  this  was  not  totally  practical  nor 
feasible.  Although  most  plants  come  under  DCAS  control,  approximately 
70  major  prime  contractor  facilities  were  retained  by  the  present  Service. 

Similarly,  a  number  of  contract  maintenance  facilities  have  been 
reassigned  to  the  parent  Service.  Although  it  was  originally  intended 
that  these  would  include  the  facilities  which  are  mainly  specialized 
contract  overhaul  operations  (i.  e. ,  engaged  solely  in  specialized  mainte¬ 
nance,  repair,  overhaul  and  modification),  this  criteria  has  not  been 
followed,  and  a  number  of  these  specialized  contractors  are  still  retained 
by  DCAS. 

For  more  consistent  and  effective  control  over  the  DoD' s  major 
specialized  repair  activities  (combined  organic  and  contract),  especially 
relating  to  consistent  over -all  quality  control  operation,  the  entire  con¬ 
tract  maintenance  program  should  be  re-examined.  Not  only  is  the 
visible  control  of  total  maintenance  by  the  SSM  essential  to  responsive 
ultimate  customer  service,  this  combined  responsibility  makes  it  pos¬ 
sible  for  the  parent  service  to  effect  a  total  maintenance  reaction,  and 
leading  to  flexibility  in  drawing  on  highly  skilled  organic  maintenance 
inspectors  to  augment  in-plant  Government  personnel  in  administer¬ 
ing/interpreting  contract  requirements  when  necessary. 


Recommendation 

It  is  recommended  that  the  Services /DCAS /OSD  follow  through  on  the 
original  concept  to  reassign  maintenance  contract  administration  to  the 
parent  Service  for  those  contractor  facilities  primarily  engaged  in 
specialized  maintenance,  repair,  overhaul  and  modification. 
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Minority  Position 


We,  the  undersigned,  nonconcur  with  the  statement  of  the  problem 
"Inspection  Cognizance  for  Contract  Maintenance,"  since  it  is  not 
germane  to  the  mission  assigned  to  Panel  Number  5. 

Some  of  the  points  raised  in  discussion  of  the  problem  seem  to 
indicate  a  need  for  the  parent  Service  to  assign  technical  guidance 
personnel  to  specific  plants  for  specific  contracts.  Such  assignments 
are  now  authorized.  In  some  instances,  when  such  assignments  are 
warranted  and  made,  economics  may  dictate  plant  assignment  to  the 
parent  Service.  Current  policy  (OSD)  makes  provision  for  such  arrange 
ments. 

Therefore,  we  do  not  concur  with  Panel  5's  presentation  on  "Inspec¬ 
tion  Cognizance  for  Contract  Maintenance.  " 


B.  C.  GERKE 
Member,  Panel  5 


S.  WILLIAMS 
Member,  Panel  5 


RELIABILITY  MANAGEMENT  RELATING  TO  MAINTENANCE 


Much  effort  has  gone  into  the  development  of  highly  sophisticated 
reliability  programs  for  select  weapons  systems  and  manned  space 
vehicles.  These  have  proven  very  effective.  Subsequently,  attention  was 
directed  to  equipments  in  every  day  use,  and  newly  designed  equipment 
built  predominantly  on  standard  design  principles,  and  from  well -tested 
parts.  Educational  programs  have  been  developed  for  reliability  manage¬ 
ment.  Focal  points'  for  reliability  management  have  been  established 
within  the  depot  organizations,  and  testing/repair  programs  oriented  to 
specific  reliability  requirements  have  been  initiated.  In  order  to  fully 
realize  the  intended  goal  of  this  program,  the  same  kind  of  discipline 
must  continue  throughout  the  life  of  the  equipment,  including  the  depot 
operation.  Substitution  of  parts  of  unknown  reliability  for  high  reliability 
component  parts  can  appreciably  degrade  equipment  reliability.  For 
effective  reliability  management,  consideration  should  be  given  to 
requiring  a  reliability  assurance  test  after  repair  that  is  consistent  with 
the  requirement  levied  on  the  new  item.  DoD  policy  on  the  relationship 
of  reliability  management  to  depot  level  operations  has  not  been  defined. 


R  e  c  omm  enda  ti  on 

It  is  recommended  that  OSD  establish  a  policy  to  provide  for  a  concerted 
effort  by  each  Service  to  develop  and  expand  the  use  of  reliability  tests 
when  reconditioning  reparable  items.  The  purpose  of  this  policy  is  to 
insure  that  replacement  items  do  not  degrade  the  quality  and  reliability 
of  the  original  production. 


142 


RELATIONSHIP  BETWEEN  RELIABILITY  ANALYSIS 
AND  PREVENTIVE  MAINTENANCE 


Present  DoD  Instructions  (3200.6/9/4100.35)  give  effective  emphasis 
to  reliability  and  maintainability  for  programs  in  the  $25  -  100  million 
R&D  projects.  Action  is  required  to  place  equivalent  emphasis  on  pro¬ 
curements  of  major  Government-furnished  equipment  items  and  compo¬ 
nents.  DoD  policies  for  reliability  and  maintainability  programs  are 
only  partially  considered  in  a  number  of  DoD  instructions;  e.  g.  ,  3200.  6  - 
Technical  Development  Plan  Preparation;  3200.9  -  Concept  Formulation; 
4100.  35  -  Integrated  Logistic  Support;  and  3232.  1  -  Maintenance  Engi¬ 
neering  Policy. 

Maintenance  engineering  analysis  (when  performed)  often  fails  to 
identify  and  document  in  precise  quantitative  terms  the  vital  product 
attributes  which  are  essential  to  effective  quality  control. 

Basically,  preventive  maintenance  and  scheduled  maintenance  are 
synonomous  (i.  e.  ,  the  sum  of  actions  performed  on  equipment  contributes 
to  uninterrupted  operation  within  design  limitations).  The  same  parallel 
may  be  drawn  for  corrective  or  unscheduled  maintenance  (i.  e. ,  the  sum 
of  actions  required  to  restore  equipment  to  operational  adequacy). 

Data  are  available  which  indicate  that  significant  amounts  of  main¬ 
tenance  work  now  being  dene  fall  into  the  unscheduled  category,  despite 
efforts  to  prepare  and  issure  complete  maintenance  work  packages. 

Work  issuances,  in  these  cases,  call  for  considerable  "open  and 
inspect"  actions  as  the  means  for  identifying  what  maintenance  should  be 
performed.  Frequently,  such  actions  do  little  more  than  expose  obvious 
surface  conditions  -  with  the  result  that  other  more  important  areas  of 
interest  are  unattended.  Effects  of  this  include  unreliable  estimates 
of  work  required  as  well  as  creating  a  void  in  the  feedback  of  essential 
reliability  data. 

Present  demands  placed  on  weapon  system  reliability  and  maintain¬ 
ability  make  it  mandatory  that  scheduled  maintenance  plans  be  available 
at  the  start  of  maintenance  operations.  This  can  only  be  done  through 
meaningful  reliability  analysis. 
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Recommendation 


It  is  recommended  that  OSD  issue  a  policy  instruction  that  makes 
specific  reference  to  the  working  relationship  between  quality  assurance 
and  reliability  and  maintainability  and  it  should  clarify  and  consolidate 
requirements  now  appearing  in  DoD  Instructions  3200.  6,  3200.9, 

4100.  35  and  3231.  1  to  include  all  reparable  systems,  components  and 
equipments  procured  by  the  DoD.  This  policy  should  further  emphasize 
the  importance  of  more  active  participation  of  capable  quality  control 
personnel  in  the  normal  process  of  work  planning,  from  design  through 
ultimate  fabrication  end  test. 
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RECONDITIONING  STANDARDS 


DoD  policy  covering  maintenance  reconditioning  ic  inadequate.  Under 
the  current  program.  Service  maintenance  operations  vary  greatly  - 
from  complete  rebuild,  i.  e. ,  restoring  equipment  to  "like-new"  condition, 
to  minimum  essential  repair  which  result  merely  in  correcting  operating 
deficiencies  on  fairly  aged  equipment.  This  is  in  part  due  to  basic  differ¬ 
ences  in  Service  policies. 

The  Services  have  published  independent  and  uncoordinated  repair 
Standards  for  similarly  maintained  equipments.  This  is  also  typical 
within  a  Service  (e.  w. ,  Army),  when  experience  has  shown  that  the  TM 
series  of  publications  is  inadequate  for  contractual  repair  purposes. 

Recognizing  its  problem,  the  Army  has  published  guidance  to  its 
commodity  Commands  (AMC  Reg  310-23  and  750-7)  for  development  and 
publication  of  maintenance  reconditioning  standards.  However,  more 
specific  policy  and  instructions  for  more  uniform  DoD-wide  program  is 
still  required. 


R  e  c  omm  enda  tions 


It  is  recommended: 

1.  That  DoD  consider  development  of  a  more  uniform  policy 
relating  to  reconditioning  standards. 

2.  That  DoD  issue  procedural  guidance  to  all  DoD  commodity 
managers  for  the  development  of  maintenance  reconditioning  standards. 
Such  an  instruction  would  assign  responsibility  for  development  of  such 
standards,  establish  a  standard  format  (v/hich  would  require  acceptable 
wear  tolerances  on  all  critical  characteristics),  and  would  be 
Service/industry  coordinated  and  acceptable  for  commercial  and  Govern¬ 
ment  repair. 


QUALITY  STANDARDS  FOR  MAS  AND 
GRANT  AID  MATERIEL 


Frequent  reports  from  recipient  countries  of  the  Grant  Aid  and 
Military  Assistance  Sales  Program  indicate  that  materiel  being  supplied 
lacks  quality,  uniformity,  contains  shortages,  and  is  not  in  conformance 
with  the  Sales  Agreement. 

There  is  often  controversy  between  the  shipping  depot  and  recipient 
country  regarding  the  requirements  of  the  materiel.  This  results  from 
the  fact  that  although  the  reconditioned  materiel.  This  results  from  the 
fact  that  although  the  reconditioned  materiel  conforms  to  the  established 
reconditioning  standards  for  the  selling  Service,  it  does  not  conform  to 
the  requirements  of  the  buying  country. 

These  conditions  exist  because  quality  standards  and  regulations 
for  MAS  and  Grant  Aid  Materiel  are  ambiguous,  lack  uniformity,  and 
are  not  sufficiently  definitive  to  assure  that  material  supplied  is  of  a 
quality  level  which  meets  Military  Sales  Agreements  and  Grant  Aid 
requirements . 


Recommendations 


It  is  recommended: 

1.  That  pertinent  and  specific  DoD  regulations  be  published 
establishing  basic  requirements  for  all  materiel  destined  for  Military 
Assistance  Sales  Program  and  the  Grant  Aid  Program. 

2.  That  detailed  and  specific  directives  be  issued  within  each 
Service  to  assure  that  materiel  destined  for  MAS  and  Grant  Aid  conform 
with  the  Sales  Agreement  negotiated  between  the  buying  country  and  the 
selling  Service. 


IMPROVED  RECONDITIONING /SERVICEABILITY 
STANDARDS  FOR  ATTRIBUTE  CHARACTERISTICS 


Reconditioning  and  Serviceability  Standards  use  such  vague  terms 
for  attribute  characteristics  (e.  g. ,  excessively  pitted,  excessively 
scratched,  excessive  roughness,  discolored,  etc)  that  the  individual 
repairing/maintaining  or  inspecting  the  completed  work  is  shouldered 
with  the  responsibility  of  quantifying  these  characteristics  on  his  own. 

This  is  a  major  problem  and  not  exclusively  that  of  maintenance. 
Vague,  generally  stated  requirements  of  this  nature  consume  tremendous 
manhours  in  attempting  to  reconcile  differences  in  opinion  and  interpreta¬ 
tion  of  intent.  As  interpretive  characteristics,  only  after  many  thousands 
of  hours  of  experience  and  discussion  is  agreement  generally  reached, 
and  then  only  on  an  individual  installation  or  facility  tisi3.  Illustrative 
of  the  problem  (and  typical)  is  a  sampling  of  maintenance/ inspection 
characteristics  taken  from  the  first  32  pages  of  a  maintenance  manual 
for  a  gun  fire  control  systems: 

1.  Scrupulously  clean. 

2.  Excessive  vibration. 

3.  Evenly  spaced. 

4.  Should  line  up. 

5.  Clean  if  necessary. 

6.  Inspect. 

7.  Should  be  uniform. 

8.  Should  be  approximately. 

9.  Replace  if  worn. 

10.  Replace  if  condition  warrants. 

11.  Will  depend  on  weather  conditions. 

Such  stated  requirements  usually  indicate  the  lack  of  regard  for  the 
later  confusion  that  may  result,  and  for  the  more  important  reason  that 
the  responsibility  for  making  the  ultimate  judgment  (and  hence  establish¬ 
ing  the  actual  standard)  is  being  passed  onto  the  worker  and  the  inspector. 

For  measurable  par^m  ters,  it  is  inexcusable  if  maintenance  stand¬ 
ards  do  not  specify  quantified  acceptable  tolerances /allowances,  partic¬ 
ularly  where  d.fccts  have  critical  (MIL-5TD-105  definition)  application. 
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Where  the  only  choice  that  exists  is  to  treat  characteristics  on  a  sub¬ 
jective  or  attribute  basis,  maximum  effort  should  be  made  bv  standards 
groups  to  develop  more  discriminating  comparison  standards,  interpre¬ 
tive  guides,  including  photographs,  etc.  .wherever  possible.  It  is  obvious 
that  this  problem  cannot  be  resolved  over  night.  However,  a  very  major 
effort  in  this  area  is  long  overdue.  Photographic  and  printing  techniques 
have  advanced  to  the  point  that  a  reasonable  effort  to  expand  on  a  DoD- 
wide  basis  on  what  has  already  been  done  in  this  area  (e.  g. ,  soldering 
guides  -  Navy  Avionics  QWS -1000a  and  NAVWEPS  00-15PA-1)  can  pay 
real  dividends. 


Recommendations 


It  is  recommended: 

1.  That  OSD  issue  a  firm  policy  (letter)  that  acknowledges  the 
need  for  increased  effort  in  developing  comparison/interpretive  standards 
for  use  in  distinguishing  attribute  characteristics  such  as  color,  fit, 
pitting,  finish,  damage,  etc. 

2.  That  DoD  take  more  aggressive  action  to  bring  the  many  indi¬ 
vidual  Service  and  Command  efforts  together  to  evolve  standards  for 
DoD-wide  use.  Needed  is  a  DoD  designated  activity  to  function  as  a 
clearing  house  and  coordinating  agency. 

3.  That  DoD/Industry  committee  be  established  to  determine  what 
standards  exist,  and  to  identify  those  which  should  be  established, 
encourage  industry  groups  to  develop  interpretive  standards  for  specific 
product  and  functional  areas,  and  monitor  the  initial  phase  of  this  pro¬ 
gram. 

4.  That. the  Services /DSA  activities  having  responsibility  for 
development  of  maintenance  reconditioning  and  quality  standards  make 
an  intense,  more  deliberate  effort  to  incorporate  quantified  toler¬ 
ances/allowances,  including  method  of  test. 
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USE  OF  CONTRACTOR  REJECTED 
MATERIAL  IN  MAINTENANCE  ACTIVITIES 


The  present  system  for  disposal  of  contractor  generated  out-of- 
specification  material  often  permits  these  materials  to  inadvertently 
enter  the  military  supply  system  for  use  by  maintenance  activities  as 
result  of  later  purchase  action. 

There  is  no  DoD  common  policy  which  would  act  as  a  control  to 
prevent  inadvertent  procurement  and  reuse  of  contractor  discarded 
material  in  our  depot/contract  maintenance  facilities.  MIL-Q-9858A 
(para  6.5)  stipulates  that  the  contractor  shall  make  known  to  the  Govern¬ 
ment  on  request  the  data  associated  with  the  costs  and  losses  in  connec¬ 
tion  with  scrap  and  with  rework  on  out-of-specification  material.  Other 
than  this  requirement,  there  is  no  standard  method  for  verifying  a 
contractor's  disposition  of  his  scrap/substandard  material. 


Recommendations 


It  is  recommended: 

1.  That  DoD  establish  a  clear,  common  policy  regarding  re-use  of 
out-of-specification  material. 

2.  That  tne  ASPR  be  amended  to  require  that  Service  purchases, 
regardless  of  quantity  or  value,  include  a  statement  to  the  effect  that 
supplies  on  order  shall  not  be  "used  or  surplus  material  unless  agreed 
to  by  the  Government,  "  thereby  placing  a  legal  as  well  as  moral  responsi¬ 
bility  on  the  supplier. 
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IMPACT  OF  CONTRACT  WAIVERS  ON  MAINTENANCE 


While  repairing  and/or  testing  materials  at  depots,  it  is  found  at 
times  that  all  characteristics  do  not  meet  tolerances  contained  in 
appropriate  specifications  as  a  result  of  contract  waivers  granted  during 
the  initital  procurement. 

After  much  time  and  effort  is  expended  by  shop  personnel  trouble 
shooting  equipment  scheduled  for  maintenance,  contact  is  sometimes 
made  with  the  appropriate  engineering  agency,  with  a  request  for 
additional  information  on  why  tolerances  exceed  specification.  At  this 
point,  it  is  occasionally  learned  that  a  waiver  was  granted  to  the  con¬ 
tractor  during  procurement  and  the  contract  specified  tolerances  were 
never  met. 

In  assessing  what  might  be  done  to  overcome  this  informational 
deficiency  on  an  interim  basis,  certain  actions  are  possible.  For 
example,  in  the  event  that  material  shipped  is  non -conforming  and  the 
waiver  did  not  require  an  official  engineering  change  notice,  some  type 
ox  permanent  documentation  might  accompany  the  equipment.  As  another 
possibility,  the  Quality  Assurance  Representative  might  include  this 
information  on  the  DD-250  shipping  document,  although  this  is  not  too 
promising  a  solution  since  the  DD-250  usually  does  not  stay  with  the 
equipment  throughout  its  life  span. 

For  a  more  lasting  correction  to  oversome  this  problem,  if  non¬ 
conformance  signficantly  affects  a  dimensional  or  test  characteristic 
of  the  product,  a  standard  DoD  form  could  be  developed  which  could  be 
affixed  to  and  accompany  the  item  or  equipment.  This  form  co''M  be 
completed  by  the  contractor,  verified  by  the  Quality  Assurance  Rep¬ 
resentative,  and  include  such  information  as:  Extent  of  noncomfox-mance, 
contractor,  contract  number,  drawing  numbers,  specification,  lot  and 
serial  numbers,  waiver  number,  approving  authority  and  date.  In  any 
event,  a  study  could  be  made  to  ascertain  the  extent  of  this  problem  in 
field  activities,  and  whether  it  occurs  frequently  enough  to  warrant 
development  of  this  type  documentation.  This  study  might  also  ascertain 
whether  the  waivers  granted  are  in  fact  affecting  characteristics  which 
in  turn  effect  the  operation  of  the  equipment. 


R  ecommendation 

It  is  recommended  that  DoD  establish  a  task  group  to  ascertain  in 
greater  detail  the  extent  of  this  problem,  and  develop  a  plan  of  action  and 
proposed  rationale  for  its  resolution. 
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QUALITY  CONTROL/INSPECTION  SKILLS  GAP 


It  is  a  well  recognized  fact,  that  in  the  past  quarter  century,  the 
complexity  and  sophistication  of  much  of  our  defense  equipment  has 
increased  at  an  explosive  rate.  In  very  recent  years,  however,  it  is 
becoming  increasingly  evident  that  inspection  skills  and  testing  technology 
are  not  keeping  pace  -  in  fact,  are  falling  behind  at  an  alarming  rate. 

This  problem  is  not  unique  to  maintenance  inspection,  but  applies 
generally  to  the  inspection  field,  and  to  a  larger  degree,  to  equipment 
operators,  users,  and  servicing  personnel  (e.  g. ,  Maintenance  Special¬ 
ists).  Moreover,  the  problem  of  lagging  inspection  skills  is  further 
aggravated  by  the  fact  that  a  program  designed  to  foster  development  of 
new,  advanced  testing  methodology  is  generally  lacking.  Since  these 
are  inter -dependent  and  play  such  an  important  role  in  the  maintenance 
area,  they  represent  a  major  maintenance  quality  control  problem. 

Only  recently  exotic  processes,  e.  g. ,  microminiturization,  electron 
beam  welding,  electro-chemical  machining,  high  energy-rate -forming, 
laser  welding,  diffusion  bonding,  honey -combing,  electro-chemical 
bonding  and  hydro-static  extrusion  represented  production  technology 
now  considered  rather  commonplace.  Electronic  gear  of  greatly 
increased  complexity,  laser  applications,  high  stress/high  temperature 
materials,  etc.  .reflect  but  a  few  of  the  material  developments  that  have 
been  responsible  for  some  of  the  unusual  and  more  challenging  demands 
on  our  inspection/testing  program. 

Generally,  the  scientist  and  engineer  have  teamed  up  in  an  effort 
to  advance  product,  without  proportionate  attention  to  upgrading 
inspection  skills  and  new  testing  technology.  Lack  of  a  parallel  and 
proportionate  attention  to  the  problems  of  inspection  and  testing  causes 
the  lag  to  become  more  pronounced  with  each  passing  year.  For  example, 
stress  limits  and  environmental  demands  have  led  to  exotic  materials 
with  unbelievable  stress -temperature  performance.  However,  conven¬ 
tional,  present  day  material  testing  (e.  g. ,  pull  type  tensile  test  methods 
and  conventional  stress  measuring  techniques)  are  no  longer  adequate 
for  these  new  high  stress -temperature  materials.  It  is  no  longer  adequate 
to  induce  the  failures  of  a  few  items,  take  average  results  and  apply  a 
'factor  of  safety. Test  methods  which  are  dynamic  and  assess  internal 
molecular  force-changes  are  needed.  We  need  to  know  the  crystalline 
molecular  condition  of  solid  rocket  motors,  not  that  separation  has  or 
had  not  occurred.  Laminations,  sandwich  construction,  special  bonding 
methods,  etc.  ,all  tend  to  point  up  the  antiquity  of  present  methods.  But 
nothing  really  exists  to  take  their  place,  and  little  has  been  done  to  fill 
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this  void.  Radiographic  techniques,  ultrasonics,  eddy  current,  and  other 
longstanding  techniques  are  being  improved,  but  with  possible  exception 
of  infra-red  (applications  for  detection  micro-electronic  defects)  no 
really  new  methods  have  appeared  on  the  inspection/testing  scene. 

Inspectors  are  frequently  placed  in  the  position  of  competing  with 
engineers  and  scientists.  In  the  competition  to  advance  our  defense 
equipment  technologically,  highly  technical  skills  are  often  unavailable 
to  inspection  to  meet  the  demands  relating  to  the  interpretation  of 
important  test  results.  Efforts  to  simplify  use  or  tests  of  "next  genera¬ 
tion"  equipment  often  suffers  because  of  the  shortage  of  technical  skills 
to  perform  the  immediate  tests  and  interpret  results.  There  is  ample 
evidence  of  this  in  such  present-day  equipment  design  programs  as 
reliability,  maintainability,  availability,  etc.  These  are  terms  that  have 
become  a  part  of  the  scientist/ engineers  drive  to  achieve  equipment 
simplicity.  The  paradox,  however,  is  that  only  too  often,  relatively 
little  is  really  done  to  relieve  the  inspector  of  an  interpretive  responsi¬ 
bility  resulting  from  the  failure  to  prescribe  clear,  meaningful,  discrim¬ 
inating  maintainability  tests,  reliability  tests,  etc. 

Traditionally,  R&D  dollars  concentrate  on  end  item  equipment. 
Occasionally,  and  usually  more  by  accident  than  design,  limited  budget 
dollars  are  diverted  to  the  problem  of  developing  testing  equipment 
specially  designed  for  use  in  testing  new  products.  This  is  the  exception, 
not  the  rule,  however,  and  equally  complex  test  equipment  is  not  the 
answer  unless  it  relieves  the  inspector  of  an  interpretive  role. 

Self  checking  equipment  (e.  g. ,  the  USAF's  VATE,  Versatile  Auto¬ 
matic  Testing  Equipment),  automatic  and  built-in  self  checking  equipment, 
and  computerized  production  control  methods  represent  a  partial  solution 
to  this  problem.  But  too  often,  deficiencies  or  malfunction  of  these 
equipments  placed  a  similar  burden  on  inspection. 

Training  of  inspectors  and  quality  control  personnel  in  our  Service 
School  has  been  intensified  in  the  past  10  years.  Although  training  is 
needed,  this  is  not  the  answer  to  solving  the  equipment-man  competance 
gap.  The  resource  and  effort  required  to  technologically  update  the  man 
at  the  same  pace  as  the  equipment  cannot  possibly  be  overcome  by  train¬ 
ing  alone. 


R  e  c  omm  enda  ti  ons 


It  is  recommended: 

1.  Thi>t  service  programs  devote  increased  attention  to  developing 
special  test  equipment  which  relieves  the  burden  on  the  inspector  and 


152 


replaces  subjective  human  evaluation  with  greater  mechan'eal  accuracy 
and  objectivity. 

2.  That  future  DoD  defense  equipment  budget  begin  to  take  account 
of  the  need  for  dollars  for  specialized  test  equipment.  This  does  not 
mean  more  complex  test  equipment,  rather  that  the  Services  should 
undertake  a  joint  program  for  the  development  of  more  adaptive  and 
perceptive  destructive  and  non-destructive  test  methods. 

3.  That  since  equipment  sophistication  is  felt  principally  by  user 
and  maintenance  activities,  an  aggressive  program  be  undertaken  to 
establish  a  quantitative  means  for  elating  (equating)  equipment,  combat 
effectiveness,  equipment  sophistication,  and  skills  levels,  as  a  partial 
means  for  establishing  optimum  design. 

4.  That  increased  emphasis  be  placed  by  all  Services /DSA  on 
quantified,  meaningful  requirements  for  equipment  reliability  and 
maintainability,  including  suitable  demonstration  provisions. 
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REDUCING  MAINTENANCE  WORKLOAD 
THROUGH  MORE  COMPLETE  "USE"  INSTRUCTION 


Lack  of  adequate  knowledge  of  the  capabilities  and  limitations  of 
equipment  by  the  user  results  in  misuse,  causing  premature  failure  or 
deterioration  of  quality. 

Normally  this  is  considered  an  "operating"  manual  problem,  and 
it  is  recognized  that  one  aspect  of  the  solution  is  to  include  equipment 
capability  information  in  operating  manuals.  This  is  not  the  complete 
solution,  however,  to  overcome  deficiencies  frequently  resulting  in 
increased  maintenance  workload  caused  by  lack  of  equipment  knowledge. 
Lack  of  knowledge  usually  reflects  inadequate  training,  but  it  is  believed 
that  improvement  of  the  operator's  manual  alone  is  not  sufficient  to 
overcome  the  problem  created  by  an  influx  into  the  supply  system  of 
newer  and  more  complex  items  for  use  by  the  rank-and-file  fighting 
man.  The  gravitation  of  more  complex  equipments  downward,  the  need 
to  draw  on  lower  skilled/IQ  people  to  operate  a  more  complex  spectra  of 
fighting  equipment,  and  a  human  reluctance  to  "take  the  time"  to  read 
operator  use/maintenance  publications  necessitates  that  we  look  to  other 
ways  to  aid  in  solving  this  problem. 

It  is  generally  accepted  that  maintenance  workload  is  only  too  often 
the  result  of  misuse,  rather  than  use.  The  best  quality  effort  (design 
and  conformance)  cannot  overcome  the  problem  of  misunderstanding, 
misuse  or  abuse.  Too  often  equipment  identified  for  overhaul  and  repair 
indicates  that  the  failure  was  not  due  to  wearout  or  normal  life  expec¬ 
tancy,  but  to  improper  field  use.  To  overcome  this  costly  overhaul 
repair  burden,  certain  actions  are  considered  appropriate. 

To  minimize  user  misuse  requires  personal  contact  concerning 
equipment  capabilities  and  limitations.  Manuals  should  be  revised  to 
contain  additional  information  where  appropriate,  but  is  is  also  necessary 
to  recognize  when  best  to  augment  manual  improvement  with  equipment 
indoctrination  of  personnel  from  Command  through  operating  levels. 

One  major  Service  activity  felt  this  particular  problem  with  new/mod¬ 
ified  items  to  be  of  sufficient  importance  as  to  warrant  special  attention. 
Accordingly,  it  approached  the  problem  with  considerable  success  by 
organizing  and  conducting  a  New  Equipment  Training  (NET)  program, 
consisting  of; 

1.  Material  Introductory  Letters  (MILs)  to  Commanders  with 
special  or  unique  information,  including  any  special  skill/operating 
requirements  for  items  prior  to  entry  into  the  supply  system. 
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2.  Analysis  of  MOS  inventory  requirements. 

3.  Development  of  appropriate  NET  packets. 

4.  Furnishing  New  Material  Introductory  Briefing  Teams  to 
field  Commanders,  when  appropriate. 

5.  Conducting  NET  courses  for  key  training  personnel. 


Recommendations 


It  is  recommended: 

1.  That  an  OSD  policy  be  issued  covering  the  use  of  New  Material 
Introductory  Briefing  Teams. 

2.  That  the  Services/DSA  exchange  information  relating  to  this 
program. 
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MAINTENANCE  TECHNICAL  DATA 


Technical  data  furnished  to  maintenance  activities  (contract,  depot, 
organization)  lack  effective,  detailed  quality  assurance  provisions,  with 
the  effect  that  quite  often  quality  production  cannot  be  assured.  Regard¬ 
ing  engineering  drawings,  the  problem  is  not  uniform.  Drawings  are 
often  found  to  be  inadequate,  and  frequently  cannot  be  obtained,  or 
require  considerable  effort. 

The  basic  deficiency  of  technical  data  (T/Os,  manuals,  etc)  stems 
from  the  lack  of  a  section  and/or  sufficient  detail  devoted  to  quality 
assurance.  While  a  requirement  for  inspections  usually  exists,  specifics 
relative  to  the  method  of  test,  number  to  be  tested  and  detailed  instruc¬ 
tion  as  to  just  how  the  item  will  be  tested  and  what  it  will  be  tested  for 
is  not  usually  provideef.  This  is  fundamentally  a  problem  stemming 
from  the  format  used,  and  could  be  corrected  by  bringing  the  format  of 
technical  manuals  more  closely  into  line  with  that  used  for  specifications. 

The  problem  relative  to  engineering  drawing  inadequacies  is  univer¬ 
sal  and  has  no  specific  domain  in  the  maintenance  area.  Deficiencies 
range  from  errors  of  omission  to  errors  of  commission.  The  question 
of  just  what  should  appear  on  an  engineering  drawing  as  opposed  to  other 
technical  data  items  constitutes  the  core  of  the  matter. 

Typical  of  deficiencies  noted  for  drawings,  and  applicable  at  least 
in  part  to  technical  manuals,  include: 

1.  Tolerances,  duplicated  or  missing. 

2.  Basic  dimensions,  duplicated  or  missing. 

3.  Dimensions  specified  to  "Points  in  Space"  and,  as  such,  not 
capable  of  verification. 

4.  Disagreements  between  notes  and  dimensions/tolerances. 

5.  Lack  of  clarity  of  notes. 

6.  Lack  of  field  tolerances. 

7.  Disregard  of  tolerance  accumulations. 

8.  Elaborate  details  and  artistry  when  simple  line  drawings  would 
be  more  effective  and  economical. 

Traditionally,  the  drawing  was  expected  to  provide  merely  the 
knowledge  of  shape  and  form.  Added,  have  been  all  or  part  of  the 
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characteristics  just  discussed,  with  little  or  no  predictable  pattern. 
Considering  the  sheer  weight  of  verification  responsibility  in  today's 
manufacturing  or  maintenance  environment,  the  need  for  resolving  this 
problem  is  readily  apparent.  Until  there  is  collateral  agreement  as 
to  the  standard  format  of  an  engineering  drawing  versus  other  consider¬ 
ations  of  "technical  data  packages"  little  improvement  can  be  expected. 


Recommendations 


It  is  recommended: 

1.  That  the  DoD  standardize  the  format  of  Maintenance  Technical 
Manuals  by  bringing  them  more  clof  2ly  into  line  with  specifications 
relative  to  quality  assurance.  Strong  formal  quality  assurance  provi¬ 
sions  similar  to  those  provided  by  specifications  would  contribute 
immensely  to  improving  the  adequacy  of  these  manuals. 

2.  That  a  comprehensive  inter-service  evaluation  be  made  to 
ascertain  which  Service  program  is  proving  most  effective  in  getting, 
maintaining,  and  issuing  engineering  drawings  which  are  generally 
adequate  from  a  maintenance  quality  assurance  viewpoint.  The  infor¬ 
mation  obtained  from  this  study  should  be  used  to  improve  Mil-D-1000 
(Engineering  Drawings  and  Associated  Lists)  and  related  Quality 
Control  Directives. 


DEFICIENCY  DATA  FEEDBACK 


The  importance  of  quality  and  reliability  feedback  data  systems  is 
generally  recognized  throughout  Government  and  industry.  DoD  Direc¬ 
tive  4155.  11,  "Improved  Management  for  Quality  and  Reliability  Assur¬ 
ance  of  Materiel,  "  highlights  the  need  for  compiling  and  using  this 
quality  information  in  Procurement,  Storage,  Maintenance  and  Field 
Service;  however,  there  continue  to  be  significant  gaps  in  obtaining  and 
properly  using  th6se  vital  data. 

Maintenance  data  in  existence  throughout  the  DoD  were  usually 
designed  for  organizational  and  field  maintenance.  Although  much  of 
the  data  generated  by  these  systems  is  useful  for  such  activities  as 
adjusting  inspection  intervals /parts  consumption  data,  the  data  gener¬ 
ated  are  usually  not  adequate,  nor  were  they  intended  for  use  in  feeding 
back  dificiency  information. 

In  a  recent  study  on  deficiency  data  feedback,  it  was  observed  that 
on  the  average  2.4  months  elapsed  from  time  the  .ailures  (deficiencies) 
were  oostrved  until  the  reports  were  received  at  the  source  production 
activity.  Assuming  the  discrepancy  to  be  production  induced,  this  time 
delay  may  well  have  permitted  large  numbers  of  defective  items  being 
delivered  to  service  organizations.  With  a  lack  of  timely,  pertinent  and 
definitive  deficiency  information,  management  action  to  provide  the 
climate  for  low  cost -high  quality  production  is  hampered. 

In  addition,  there  are  several  other  causes  for  concern  over 
adequacy  of  the  deficiency  feedback  reporting  systems  such  as: 

1.  Is  deficiency  reporting  considered  wasted  time? 

2.  Is  there  lack  of  payoff  to  the  reporting  activity,  i.  e.  ,  lack 
of  evidence  of  corrective  or  preventive  action? 

3.  Is  the  effort  too  costly? 

4.  Is  there  failure  to  follow  established  systems? 

5.  Lack  of  objective  and  procedural  guidance  on  what  to  report, 
i.  e.  ,  Minor  or  relatively  inconsequential  vs  signiticant  deficiencies. 


Recommendations 


It  is  recommended: 

1.  That  OSD  initiate  action  to  examine  the  existing  deficiency 
reporting  systems  to  determine  which  of  these  will  provide  the  most 
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complete  and  timely  feedback  of  quality  control  deficiency  data.  This 
system  should  then  receive  priority  attention  for  service -wide  implemen¬ 
tation. 

2.  That  OSO  take  action  to  address  the  attendant  major 
problem  of  coordinating  the  data  feedback  systems  required  by 
DoDD  3232.  1  -DoD  Maintenance  Engineering  Program,  DoDD 
4100.  35  -  Development  of  Integrated  Logistic  Support  for  Systems  and 
Equipments,  DoDI  7220.  14  -  Uniform  Cost  Accounting  for  Depot  Mainte- 
ance,  DoDI  7730.25  -  Equipment  Distribution  and  Condition  (EDAC)  - 
Statistical  Reporting  System,  Proposed  DoD  -  Configuration  Accounting 
and  Proposed  DoD  -  Resources  Management;  and  to  place  these  actions 
in  harmony  with  the  intent  of  DoDD  5000.  11  -  Data  Elements  and  Data 
Codes  Standardization  Program  and  DoDI50u0.  12  -  Data  Elements  and 
Data  Codes  Standardization  Procedures  as  they  pertain  to  data  element 
s  tandardization. 


REPORT  OF  PANEL  6 


TIT  LE:  Reliability  and  Maintainability  Assessment 
OBJECTIVE: 

To  recommend  actions  essential  to  the  quantitative  assess¬ 
ment  of  materiel  reliability  and  maintainability  during  all 

phases  of  its  life  cycle. 

TOPICS  DISCUSSED: 

1.  Consistent  Assessment  Criteria  for  ReliabiJity/Main- 
tainability 

2.  Reliability/Maintainability  Prediction  Techniques 

3.  Test  Programs  as  an  Input  to  Reliability/M.aintainability 
Assessment 

4.  Field  Data  Feedback 

5.  Reliability  and  Maintainability  Data  Storage  and 
Retrieval 

6.  Research  in  and  Validation  of  Reliability  and  Maintain¬ 
ability  Assessment  Techniques 

7.  Use  of  Reliability  and  Maintainability  Assessment 
Results  by  Management 

8.  Interservice  Coordination 
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CONSISTENT  ASSESSMENT  CRITERIA  FOR 
RELIABILITY /MAINTAINABILITY 


If  the  DoD  reliability  and  maintainability  assessment  program  is  to 
be  an  effective  part  of  total  program  management,  it  mast  provide 
decision  makers  with  consistent  measures  of  reliability  and  maintain¬ 
ability.  A  review  of  the  assessment  efforts  currently  under  way  in  all 
three  departments  resulted  in  two  conclusions.  First,  with  some 
exceptions  (such  as  strategic  delivery  systems  where  JCS  and  OSD  have 
imposed  detailed  requirements)  the  departments  do  not  apply  consistent 
measures  of  reliability  and  maintainability  for  systems.  This  lack  of 
consistency  is  not  readily  apparent  but  is  revealed  in  differences  in  what 
is  meant  by  a  reliable  system.  The  second,  and  even  more  serious 
problem,  is  the  lack  of  consistency  across  the  life  cycle.  This  is 
usually  manifest  in  the  form  of  restricted  definitions  of  a  reliable  system 
during  early  development  which  are  gradually  replaced  by  more  realis¬ 
tic  definition  of  system  reliability  as  the  system  is  deployed. 

Panel  6  reviewed  the  following  documents  to  determine  if  DoD  policy 
guidance  in  this  area  could  be  improved:  DoD  Instruction  3200.  6,  DoD 
Directive  3200.9,  DoD  Directive  4100.35,  and  DoD  Directive  4155.11. 
From  the  point  of  view  of  an  adequate  assessment  program,  it  was  con¬ 
sidered  essential  that  DoD  policy  meet  the  following  criteria: 

1.  Reasonable  consistency  among  the  departments, 

2.  A  consistent  basis  for  assessment  throughout  the  life  cycle,  and 

3.  A  means  of  relating  measured  or  predicted  reliability  and  main¬ 
tainability  to  one  another  or  to  other  parameters  effecting  overall  system 
mission  performance. 

From  this  review  it  is  clear  that  the  departments  have  issued  or 
will  issue  implementing  directives  requiring  life  cycle  assessment, 
however,  it  is  also  clear  that  the  current  departmental  directives  fail 
to  meet  all  of  the  above  criteria. 

Although  there  is  a  DoD  goal  of  achieving  an  optimal  balance  between 
system  effectiveness,  performance,  schedule,  and  total  cost,  there  is  no 
clear-cut  DoD  policy  that  requires  a  comprehensive,  integrated,  single 
thread  effectiveness  evaluation  program  throughout  all  phases  of  a 
weapon  system  life  cycle.  Since  system  effectiveness  analysis  can  be  a 
vital  and  dynamic  decision-making  tool,  it  can  provide  greater  insight 
into  such  areas  as: 
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1.  Establishing  attainable  requirements. 

2.  Probability  of  mission  success. 

3.  Isolation  of  problem  areas. 

4.  Ranking  the  criticality  of  problems. 

5.  Evaluation  of  alternative  solutions. 

An  analysis  of  pertinent  DoD  Directives  provide  the  following: 

3200.  6  (Reporting  of  Research,  Development  and  Engineering 
Program  Information)  requires  consideration  of  reliability  and  main¬ 
tainability  in  the  preparation  of  requirements  documents  and  technical 
development  plans.  It  is  a  permissive  document  in  that  it  says  .  .  due 
consideration  shall  be  given  to  all  characteristics,  including  reliability 
and  maintainability,  in  the  early  planning  and  feasibility  study  stages, 
and  comprehensive  reliability  and  maintainability  programs  are  expected 
for  operational  development  projects.  It  is  intended  that  both  the  human 
and  hardware  aspects  of  reliability  and  maintainability  be  considered. 

The  goal  is  a  balanced  and  integrated  effort  aimed  at  optimizing  opera¬ 
tional  effectiveness,  total  cost  and  early  availability.  "  The  requirement 
is  weak,  doesn't  start  early  enough;  e.  g.  ,  should  start  with  Advanced 
Development  Objectives  (ADO's)  and  is  not  applicable  beyond  the  defini¬ 
tion  phase.  There  are  no  provisions  for  passing  information  (data)  or 
analysis  (models)  into  acquisition. 

3200.9  (Initiation  of  Engineering  and  Operational  System  Develop¬ 
ment)  is  the  only  DoD  Directive  that  addresses  the  total  system /cost 
effectiveness  evaluation.  It  does  not  provide  a  framework  or  even 
ground  rules  for  the  conduct  of  system  effectiveness  evaluation  thus 
comparisons  between  system  and/or  trade-off's  within  systems  are 
difficult  if  not  meaningless.  It  addresses  only  the  concept  formulation 
and  contract  definition  phases.  It  does  not  provide  for  passing  to  the 
acquisition  phase  that  data  and  analysis  performed  for  contract  defini¬ 
tion  nor  does  it  require  a  later  audit  to  determine  or  compare  hardware 
performance  with  contract  definition  base  line  prediction. 

4100.  35  (Development  of  Integrated  Logistic  Support  for  Systems 
Equipments)  is  of  concern  to  Panel  6  in  that  it  requires  the  use  of 
reliability  and  maintainability  data  and  analysis  along  with  total  system 
analysis  in  the  implementation  of  the  integrated  logistic  concept.  Addi¬ 
tionally,  it  requires  logistic  matters  and  costs  to  be  considered  from 
the  trade-offs  in  feasibility  studies  and  the  concept  formulation  phase 
into  the  system  operational  phase.  It  recognizes  the  interrelationship 
of  the  functional  disciplines,  and  directs  that  they  be  mutually  supporting. 
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DoD  Directive  4155.  11  (Improved  Management  for  Quality  and 
Reliability  Assurance  of  Materiel)  treats  quality  as  a  functional  activity 
applicable  during  the  acquisition  and  operational  phase.  It  attempts  to 
include  reliability  by  use  of  the  term  in  several  places  equated  with 
quality.  In  fact,  one  could  strike  the  word  "reliability"  each  place  it 
appears  and  not  affect  the  directive  in  any  way.  The  point  being,  there 
exists  a  separate  and  well  defined  reliability  program  apart  from  the 
quality  assurance  program.  The  quality  assurance  program  is  a  sepa¬ 
rate  and  distinct  program,  the  two  are  not  and  cannot  be  treated  mean¬ 
ingfully  in  the  same  directive.  Therefore,  the  term  "reliability"  should 
be  deleted  from  DoD  Directive  4155.  11. 

Thus  at  present,  there  is  no  requirement  for  cradle-to-grave  evalu¬ 
ation  of  the  overall  effectiveness  of  military  weapon  systems.  There  is 
considerable  emphasis  on  this  during  the  concept  formulation  and  con¬ 
tract  definition  phases.  However,  during  engineering  design  and  pro¬ 
duction,  there  is  no  requirement  for  continuing  evaluation  by  DoD. 
Effectiveness  evaluation  programs  are  being  conducted  by  a  few  of  our 
Defense  contractors  during  design  and  production  to  satisfy  their  inter¬ 
nal  needs.  However,  it  is  questionable  whether  these  contribute  to  the 
mainstream  of  the  system  program  or  are  little  more  than  academic 
exercises.  At  any  rate,  customer  and  contractor  objectives  cannot  be 
expected  to  be  identical;  i.  e. ,  "best"  in  a  contractor's  eyes  may  not 
be  "best"  from  a  military  viewpoint.  Interest  picks  up  somewhat  during 
the  operational  phase,  but  the  requirements  for  operational  evaluations 
vary  considerably.  Variations  occur  not  only  between  the  military 
departments,  but  even  between  the  developer  and  user  within  a  single 
department.  Major  program  changes  can  be  made  and  are  being  made 
without  benefit  of  thorough  evaluation  in  terms  of  impact  on  total 
system/cost  effectiveness.  Included  are  Engineering  Change  Proposals 
(ECP's),  changes  in  operating  or  maintenance  concepts,  changes  in 
manning  levels,  etc. 

Several  preliminary  actions  have  been  taken  by  the  separate  services. 
The  Air  Force  Systems  Command  sponsored  the  Weapon  System  Effec¬ 
tiveness  Industry  Advisory  Committee  (WSEIAC)  Study  which  provided  a 
framework  for  system/cost  effectiveness  analysis.  The  Army  (AMC) 
has  issued  AMC  Reg.  No.  70-28  which  establishes  an  AMC  Systems 
Analysis  Center  ( AMSAC)  as  well  as  requiring  system  analysis  on  major 
systems  in  the  concept  formulation  and  definition  phases.  The  Navy 
meanwhile  has  favored  a  slightly  different  framework  for  effectiveness 
analysis. 

The  Systems  Performance  Effectiveness  Group  at  the  Naval  Applied 
Sciences  Laboi-ntory  is  spearheading  the  fundamental  activity  in  this 
area. 
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The  above  analysis,  critical  comment,  and  recommendations  are 
made  to  indicate  the  absence  of  and  need  for  a  positive,  consistent 
single  thread  assessment  criteria  for  reliability  and  maintainability 
analysis  and  measurement.  The  DoD  should  insure  sufficient  coordina¬ 
tion  of  their  functional  directives  to  assure  their  compatibility,  only 
essential  overlap  or  duplication,  and  that  systems/equipments  are 
covered  from  feasibility  studies  to  phase  out  of  operations. 

The  original  intent  was  to  recommend  changes  to  existing  DoD 
directives  which  would  assure  consistency  among  the  departments;  how¬ 
ever,  the  DDRE,  I&L  dichotomy  of  directives  prohibited  this  course  of 
action. 


Recommendation 

It  is  recommended  that  OSD  issue  an  instruction  requiring  consistent 
assessment  of  system  s/equipments  in  terms  of  probability  of  mission 
accomplishment  (or  other  appropriate  measure  of  effectiveness)  during 
all  phases  of  its  life  cycle.  (A  proposed  draft  instruction  is  given  in 
Appendix  A. ) 
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RELIABILITY/MAINTAINABILITY  PREDICTION 

TECHNIQUES 


Predictions  are  the  only  input  information  available  for  reliability 
and  maintainability  assessment  during  feasibility  and  early  develop¬ 
ment.  Many  vital  and  far-reaching  decisions  are  made  based  on  com¬ 
parisons  of  predicted  reliability  for  future  equipments  versus  achieved 
reliability  for  existing  equipments.  Panel  6  carefully  reviewed  the 
state  of  the  reliability  and  maintainability  prediction  art  to  see  if  avail¬ 
able  techniques  are  sufficiently  accurate  in  light  of  the  importance  of 
the  decisions  which  they  influence  (such  as  the  decision  to  pursue  the 
development  and  production  of  a  new  system). 

Considerable  progress  has  been  made  in  this  field  in  the  last  few 
years.  Recent  improvement  actions  include  the  revision  of  MIL-STD- 
756  and  its  associated  MIL  HANDBOOK  217.  These  revisions  resulted 
in  improvement  in  the  correction  for  various  environments  and  equip¬ 
ment  types  (K  factors).  For  the  first  time,  differences  in  degradation 
by  component  type  were  recognized  and  the  procedures  were  revised  to 
provide  for  a  more  accurate  prediction  of  environmental  degradation. 

A  second  improvement  is  the  pending  publication  of  MIL  HANDBOOK 
47 2 --Maintainability  Prediction.  This  Handbook  represents  the  first 
step  toward  standardizing  maintainability  prediction  techniques. 

In  spite  of  this  progress,  much  work  remains  to  be  done.  Existing 
procedures  are  inadequate  in  terms  of  their  accuracy  and  are  limited 
in  their  application  to  certain  equipment  types.  Specific  areas  where 
improvement  is  urgently  needed  are  as  follows: 

1.  Reliability  and  Maintainability  during  Feasibility 

MIL-STD-756  outlines  procedures  for  reliability  prediction  in 
the  feasibility  and  design  steps  of  system  development.  Feasibility  pre¬ 
diction  is  implemented  by  means  of  Chart  I  of  the  Standard;  design 
prediction  by  the  data  base  of  MIL-HDBK -21 7A.  Neither  procedure,  in 
its  present  state,  is  adequate  for  today's  needs.  The  feasibility  predic¬ 
tion  procedure  requires  knowledge  of  active  element  count  and,  therefore, 
cannot  be  applied  until  the  program  approaches  the  design  phase.  Further, 
the  method  is  based  on  failure  rates  for  average  active  element  groups 
and,  in  general,  will  produce  predictions  which  are  op-imistic  for  high 
power  equipment  categories,  such  as  radar,  and  pessimistic  for  low 
power  categories,  such  as  data  processing.  The  failure  to  distinguish 
between  high  and  low  failure  rate  functions  also  deprives  the  conceptual 
phase  of  early  information  on  potential  problem  areas.  Finally,  the 
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feasibility  prediction  procedure  of  Chart  I  is  inconsistent;  the  predicted 
reliability  can  vary  considerably  depending  on  the  size  one  chooses  for 
,fa  functional  block. "  For  example,  if  one  divides  the  equipment  into 
two  "blocks,"  the  predicted  reliability  will  differ  significantly  from  that 
predicted  for  the  same  equipment  divided  into  ten  blocks.  There  is 
inadequate  recognition  of  the  effect  on  reliability  of  interconnection  and 
packaging. 

2.  Accuracy  of  Reliability  and  Maintainability  Predictions 

MIL-STD-756  and  MIL-HDBK-217A  ignore  the  existence  of  the 
degradation  due  to  environments  other  than  electrical  and  thermal  stress 
and  hence  result  in  optimistic  estimates  of  reliability.  Various  users  of 
this  standard  are  having  to  develop  their  own  K  factors  or  use  other 
techniques  to  get  realistic  predictions.  A  major  effort  to  correlate  pre¬ 
dictions,  R&D  test  results  and  measured  reliability  with  a  view  to 
deriving  realistic  K  factors  appears  warranted.  A  further  deficiency  is 
the  fact  that  the  failure  rates  lump  both  catastrophic  failures  and  param¬ 
eter  change  failures.  This  lumping  process  tendt  not  only  to  reduce 
precision  of  prediction,  but  also  to  restrict  the  utility  of  the  data  as 
design  guidance  criteria.  Catastrophic  failures  are  theoretically  subject 
to  control  through  quality  control.  Drift  characteristics  may  be  compen¬ 
sated  for  through  circuit  design  which  establishes  level  of  tolerance  to 
part  parameter  change.  Without  a  breakout  of  data  on  each,  catastrophic 
failure  rate  versus  stress  and  part  parameter  change  versus  time,  con¬ 
trol  procedures  cannot  be  effectively  implemented. 

3.  Keeping  Reliability  Data  Current 

The  second  factor  is  that  the  MIL-HDBK  is  not  keeping  pace 
with  the  state  of  the  art,  e.  g.  ,  integrated  circuits  are  not  adequately 
covered.  More  emphasis  is  needed  and  positive  action  taken  to  keep 
the  handbook  as  current  as  possible. 

4.  Inadequate  Coverage  of  Mechanical  Equipment 

Adequate  mechanical  and  electromechanical  equipment  predic¬ 
tion  techniques  are  nonexistent.  Military  documentation  has  not  kept 
pace  with  techniques  developed  and  applied  by  industry.  Currently, 
both  MIL-HDBK-217A  and  MIL-HDBK -472  are  inadequate  with  respect 
to  covering  non-electronic  equipment.  Information  on  failure  rates  and 
failure  modes  on  mechanical  equipment  is  extremely  scarce  in  MIL- 
HDBK  -VIA,  Likewise,  repair  time  data  tor  mechanical  equipment 
necessary  to  implement  maintainability  prediction  is  totally  lacking  in 
MIL-HDBK  -472. 

5.  Maintainability  Prediction  Data  Base 

In  the  area  of  maintainability  prediction,  there  are  four  tech¬ 
niques  developed  and  contained  in  MIL-HDBK -472.  The  data  base  upon 
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which  these  techniques  are  based  is  outdated,  the  techniques  have  been 
developed  exclusively  in  the  ground,  shipboard,  and  airborne  electronics 
environment,  and  thus  the  techniques  can  be  easily  misapplied.  There 
is  general  agreement  that  we  do  not  have  adequate  means  for  predicting 
maintainability  parameters,  and  particularly  so  in  the  earlier  life  cycle 
phases  when  hardware  is  not  well  defined. 

6.  Prediction  of  Reliability  for  "Non-Time  Dependent"  Devices 

MIL-STD-756  and  MIL-HDBK -21 7A  do  not  cover  reliability 
prediction  for  "non-time  dependent"  devices.  Certain  material  like 
missile  handling  devices,  switching  equipment, etc.  ,  are  cyclic  in  their 
operation  rather  than  time  dependent.  For  some  equipment  such  as 
certain  missiles,  environmental  stresses  far  exceed  time  as  a  force 
influencing  failure.  New  and  more  accurate  reliability  prediction  tech¬ 
niques  for  such  equipment  are  needed. 

7.  Summary 

The  following  summarizes  the  review  of  the  MIL  documents 
discussed  in  the  paragraphs  above. 

a.  MIL-STD-756A  -  Reliability  Prediction,  15  Mav  1963. 
Establishes  procedures  for  predicting  the  quantitative  reliability  of 
aircraft,  missiles,  satellites,  electronics  equipment  and  their  subdivi¬ 
sions  for  th»  purpose  of  identifying  design  problem  areas,  and  for 
apportionment  of  reliability  requirements,  as  either  the  Feasibility 
Prediction  or  Design  Prediction. 

Deficiency:  Inadequate  in  illustrating  techniques  for  handling  functional 
relations  of  elements  comprising  a  complex  system. 

b.  MIL-HDBK -217A  -  Reliability  Stress  and  Failure  Data 
for  Electronic  Equipment,  1  December  1965.  Provides  guidance  for 
implementation  of  MIL-STD-756,  reliability  prediction;  presents 
several  reliability  prediction  techniques,  useful  at  various  stages  of  a 
program,  depending  upon  available  information;  identifies  uses  and 
limitations  of  reliability  prediction;  provider  a  list  of  reliability  infor¬ 
mation  services  where  further  data  may  be  acquired. 

Deficiencies:  Very  little  data  on  mechanical  items,  inadequate  K  factors 
for  various  environment. 

c.  MIL-HDBK -472  -  Maintainability  Prediction  (in  print). 
Gives  philosophy  of  maintainability  predication,  trade-offs  between 
reliability  and  maintainability,  costs  and  payoffs  of  predictions.  Pro¬ 
vides  several  maintainability  costs  and  payoffs  of  predictions.  Provides 
several  maintainability  prediction  procedures. 

Deficiencies:  Procedures  need  updating,  particularly  repair  time 
elements.  Procedures  applicable  to  nonelectronic  items  lacking. 
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The  above  list  is  not  intended  to  be  all  inclusive  but  illustrates  the 
fact  that  additional  work  is  needed. 


Recommendations 


It  is  recommended! 

1.  That  the  departments  and  agencies  give  high  priority  to  the  con¬ 
duct  of  reliability  and  maintainability  prediction  studies  assigned  by  the 
Reliability  and  Maintainability  Assessment  Steering  Committee.  (See 
Recommendation  under  Interservice  Coordination. ) 

2.  That  the  departments  intensify  efforts  to  correlate  test  and 
field  results  to  orginal  predictions  and  use  results  to  improve  prediction 
techniques. 
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TEST  PROGRAMS  AS  AN  INPUT  TO 
RELIABILITY/MAINTAINABILITY  ASSESSMENT 


Integrated  test  programs  conducted  throughout  development  and 
production  which  attempt  to  meet  the  test  needs  of  all  disciplines  through 
sharing  of  test  time  and  test  hardware  have  become  a  way  of  life  for 
many  major  DoD  systems.  These  programs  are  one  of  the  major  sources 
of  input  information  for  the  reliability  and  maintainability  assessment 
program. 

Normally,  those  tests  specifically  designed  for  reliability  and 
maintainability  demonstration  occur  late  in  the  development  cycle. 

Thus,  if  a  major  gap  in  the  assessment  program  is  to  be  avoided,  maxi¬ 
mum  utilization  cf  data  for  reliability  and  maintainability  must  be  made 
from  other  types  of  tests.  Unfortunately,  experience  indicates  that  much 
of  this  data  is  wasted  from  the  point  of  view  of  reliability  and  maintain¬ 
ability  assessment. 

Some  of  the  factors  affecting  this  loss  of  data  are: 

1.  Reliability  and  maintainability  personnel  do  not  always  partici¬ 
pate  in  detailed  test  planning. 

2.  Data  necessary  to  the  assessment  of  reliability  and  maintain¬ 
ability  are  not  recorded;  c.  g.  ,  elapsed  run  time. 

3.  Reliability  and  maintainability  personnel  and  test  personnel  do 
not  agree  in  advance  on  whether  or  not  test  is  valid  for  reliability  and 
maintainability  purposes  and  do  not  agree  on  the  definition  of  a  failure. 

Most  of  these  problems  can  be  overcome  if  adequate  coordination 
exists  between  reliability  and  maintainability  and  test  personnel  prior 
to  the  test.  Such  coordination  should  include  review  of  test  plans, 
adequacy  of  sample  sizes,  data  requirements,  agreement  on  test  valid¬ 
ity,  and  definition  of  failure.  Project  managers  should  assure  that  this 
coordination  takes  place. 

Reliability  demonstration  tests  are  a  vital  part  of  the  assessment 
program.  Reliability  and  maintainability  demonstration  tests  are  becor 
ing  increasingly  common  as  a  part  of  R&D  contracts  for  major  systems 
and  equipments  and  are  used  to  a  somewhat  lesser  degree  in  production 
contract*  for  these  same  systems.  However,  reliability  and  maintain¬ 
ability  demonstration  tests  of  smaller  items  are  relatively  rare.  A 
sample  of  10,000  current  contracts  administered  by  DCASR  Chicago 
revealed  that  only  28  contracts  contained  reliability  demonstration  tests. 
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This  represented  .  28  percent  of  the  number  of  contracts  and  11.7  per¬ 
cent  of  the  dollar  value.  Increased  reliability  and  maintainability 
demonstration  tests  would  substantially  increase  the  reliability  and 
maintainability  data  base  and  more  importantly  substantially  increase 
the  reliability  of  delivered  hardware. 


Recommendations 


It  is  recommended: 

1.  That  as  the  cognizant  department,  the  Air  Force  initiate  a 
change  to  MIL -STD-785  adding  a  requirement  for  coordination  of  non¬ 
reliability  tests  with  reliability  and  maintainability  personnel.  (A  pro¬ 
posed  new  paragraph  is  given  in  Appendix  B. ) 

2.  That  the  Services  establish  policy  requiring  increased  use  of 
reliability  and  maintainability  demonstration  requirements  for  more 
types  of  equipments  including  reprocurements  and  replenishmert  spares. 
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FIELD  DATA  FEEDBACK 


A  major  data  source  for  reliability /maintainability  assessment  is 
field  failure  and  maintenance  information.  Panel  6  reviewed  the  exist¬ 
ing  data  feedback  systems  with  respect  to  their  ability  to  provide  ade¬ 
quate  and  accurate  inputs  to  reliability/maintainability  assessments. 

All  three  services  have  large  reporting  systems  to  provide  failure 
data.  (Army-TAERS,  Air  Force -66-1,  Navy-MDC).  While  there  are 
differences  in  these  systems,  they  do  have  several  things  in  common: 

1.  They  all  were  basically  designed  for  maintenance  and  supply 
management  and  inventory  control  purposes. 

2.  None  of  them  are  specifically  designed  to  measure  reliability 
and  maintainability. 

3.  Inadequate  mechanisms  for  feeding  field  data  back  into  design. 

4.  All  result  in  masses  of  data  fed  to  a  central  location. 

5.  All  are  dependent  upon  the  motivation  of  the  initiator  of  the 
report  for  their  accuracy. 

6.  All  are  slow  to  respond  to  non-standard  data  outputs  which  are 
often  required  in  reliability  and  maintainability  assessment. 

To  meet  the  needs  of  reliability  and  maintainability  assessment  all 
three  departments  have  resorted  to  special  data  collection  efforts  in 
high  cost  complex  systems  to  obtain  the  necessary  reliability  and  main¬ 
tainability  data.  These  programs  consist  of  installing  additional  time 
meters  in  fielded  equipment  when  required  deploying  trained  data  col¬ 
lectors,  and  collecting  additional  elements  of  time  and  event  oriented 
data  using  specially  devised  data  forms.  Generally,  these  programs 
have  been  successful  but  suffer  from  the  obvious  disadvantage  of  rela¬ 
tively  high  cost.  While  large  projects  can  easily  afford  such  programs, 
they  cannot  effectively  be  used  for  all  equipments.  Thus,  some  type  of 
large  data  feedback  system  will  continue  to  be  required. 

A  problem  of  this  magnitude  could  not  be  solved  during  this  confer¬ 
ence.  However,  Panel  6  did  arrive  at  the  following  conclusions: 

1.  Without  exception,  the  field  data  feedback  and  analysis  systems 
of  the  departments  are  not  presently  providing  sufficiently  adequate  and 
accurate  information  for  reliability  and  maintainability  assessment. 
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2.  These  data  systems  or  their  successors  should  be  designed  to 
provide  all  of  the  data  required  for  effectiveness  assessment  to  include 
maintenance 'management  requirements. 

3.  Field  data  collection  systems  should  be  designed  to  provide 
reliability  and  maintainability  data  consistent  with  data  collected  earlier 
in  the  program  in  order  to  support  a  life  cycle  assessment  program. 

4.  In  creating  their  current  data  feedback  systems,  the  depart¬ 
ments  have  found  it  necessary  to  compromise  data  requirements  for 
various  equipment  types  in  order  to  meet  a  standard  format  goal. 

Further  consolidation  with  its  additional  "standardization"  is  undesir¬ 
able.  If  it  becomes  necessary  to  further  standardize,  then  such  action 
should  proceed  along  functional  lines;  e.  g.  ,  all  three  services  might 
use  similar  data  systems  for  aircraft,  another  system  for  missiles,  etc. 

5.  The  controlled  reliability  and  maintainability  monitoring  pro¬ 
grams  provide  the  most  accurate  present  means  of  monitoring  reliabil¬ 
ity  and  maintainability  of  equipment  during  service  tests  and  operational 
use. 


6.  The  overall  subject  of  reliability  and  maintainability  measure¬ 
ment  for  field  data  feedback  deserves  more  thorough  study.  Some  of 
the  problems  which  need  answering  are: 

a.  What  level  of  accuracy  and  adequacy  can  be  expected  from 
"census  type"  data  feedback  programs  versus  detailed  feedback  using 
sampling  techniques? 

b.  What  effect  does  use  of  data  by  the  "chain  of  command"  to 
assess  unit  effectiveness  have  on  data  accuracy? 

c.  What  are  the  limits  on  the  amount  of  data  that  can  be 
required  and  still  expect  reasonable  accuracy? 

7.  Special  data  collection  efforts  are  normally  discontinuous. 

This  results  in  the  following: 

a.  Valuable  information  necessary  to  arrive  at  proper  opera¬ 
tional  and  logistics  support  decisions  is  lost; 

b.  Reliability  and  maintainability  parameters,  established 
during  development,  cannot  be  correlated  with  like  parameters  assessed 
from  field  data  during  the  operat;onal  phase; 

c.  Vital  weapon  system  data  for  use  in  follow-on,  similar 
weapon  system  acquisition  efforts  is  lost. 
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Recommendations 


It  is  recommended: 

1.  That  the  Service  conduct  a  study  of  its  field  data  feedback  sys¬ 
tem  whose  objective  is  to  determine  those  changes  necessary  to  make  that 
system  capable  of  providing  all  of  the  data  required  for  effectiveness 
assessment  (to  include  reliability  and  maintainability)  as  well  as  for 
maintenance  management.  This  study  should  be  conducted  by  personnel 
who  are  currently  not  associated  with  the  data  feedback  system. 

2.  That  OSD  sponsor  an  "in  depth"  study  of  Lhe  filed  data  feed¬ 
back  problem  which  would  examine  such  factors  as  the  relative  accuracy 
of  special  sampling  programs  versus  "census  type"  feedback  systems, 
the  effect  of  user  motivation  and  means  of  improving  it,  centralized  and 
standardized  data  systems  versus  decentralized  and  specialized  systems, 
etc. 


3.  That  the  Services  continue  their  use  of  special  reliability  and 
maintainability  monitoring  programs,  especially  during  the  operational 
phase. 

4.  In  the  interim,  that  all  three  departments  continue  efforts  to 
improve  the  reliability  and  maintainability  coverage  of  their  respective 
field  data  feedback  systems  with  emphasis  on  accuracy  of  input  data. 
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RELIABILITY  AND  MAINTAINABILITY  DATA 
STORAGE  AND  RETRIEVAL 


The  storage,  retrieval,  and  distribution  of  data  are  activities  of 
prime  importance  in  reliability  and  maintainability  prediction,  testing, 
and  demonstration.  All  assessment  is  based  on  data  collected  from 
controlled  tests  or  operating  experience. 

As  each  of  the  services  developed  reliability  and  maintainability 
practices,  it  established  data  processing  methods  which  best  served 
it  in  the  unique  set  of  conditions  affecting  its  operations.  Some  use 
was  made  of  existing  data,  data  collection  systems  and  data  processing 
capabilities  in-house.  Separate  data  collection,  processing,  or  utiliza¬ 
tions  were  frequently  developed  to  handle  specific  programs,  weapon 
systems,  or  operational  requirements.  As  a  result,  a  large  number  of 
separate  systems  have  proliferated,  each  with  its  attendant  data  bank. 

At  present,  no  system  exists  to  correlate  data  from  one  data  bank  to 
another. 

Some  of  the  types  and  uses  of  data  sources  are  described  in 
Appendix  C.  The  listing  is  not  complete,  but  does  serve  to  indicate 
the  extent  of  variation  between,  requirements  for,  and  the  uses  of  dat$. 
It  is  also  possible  to  see  in  the  summary,  varied  manners  of  handling, 
response  times,  and  forms  of  presentation  of  retrieved  data. 

The  obvious  weakness  in  the  present  reliability  and  maintainability 
data  handling  program  is  the  near  impossibility  to  adapt  readily  the 
data  in  one  data  bank  to  the  needs  of  somebody  to  whom  it  is  "foreign.  " 
On  the  other  hand,  imposition  of  any  set  of  conditions  on  one  system  to 
conform  completely  with  another  would  lessen  that  system's  merits  for 
which  it  was  tailored.  The  time  to  implement,  cost  to  create,  and  lack 
of  efficiency  of  an  all-inclusive  data  processing  and  utilization  method 
throughout  one  of  the  services  seem  to  render  this  solution  unfeasible. 

Use  can  be  made  by  DoD  activities  of  most  data  now  stored  or  being 
collected  only  with  an  application  of  effort  and  ingenuity  on  the  part  of 
knowledgeable  analysts.  Adjustments  of  the  diversely  stored  and  coded 
data  to  a  common  "cross-talk"  may  well  be  possible.  With  direction 
from  OSD,  data  could  be  handled  and  presented  by  tjie  several  systems 
in  ways  which  would  make  the  information  in  each  bank  readily  acces¬ 
sible  to  all  concerned. 
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Recommendations 


It  is  recommended: 

1.  For  the  short  term,  that  OSD  prepare,  publish,  and  keep 
current  a  Reliability  and  Maintainability  Data  Bank  Index.  This  docu> 
ment,  preferably  in  the  form  of  a  handbook,  would  briefly  describe  all 
existing  reliability  and  maintainability  data  storage  and  retrieval  systems, 
indicate  the  types  of  data  contained  therein  and  give  specifics  on  how  to 
properly  request  needed  information.  (Appendix  C  represents  an  initial 
effort  in  this  regard. ) 

2.  For  the  long  term,  that  action  be  taken  by  the  Services  under 
OSD  guidance  to  tie  decentralized  reliability  and  maintainability  data 
banks  together  in  a  network  which  would  allow  rapid  interchange  of 
information. 


RESEARCH  IN  AND  VALIDATION  OF  RELIABILITY  AND 
MAINTAINABILITY  ASSESSMENT  TECHNIQUES 


Today's  reliability  and  maintainability  assessment  programs  are 
conducted  using  tools  and  techniques  developed  in  the  past  several  years. 
Continued  improvement  in  the  existing  "tools  of  the  trade"  as  well  as  the 
development  of  new  tools  are  essential  if  reliability  and  maintainability 
assessment  programs  are  to  keep  pace  with  advancing  technology  and 
management  needs.  Panel  6  reviewed  the  reliability  and  maintainability 
assessment  field  and  identified  a  number  of  areas  where  research  is 
required.  These  are  as  follows: 

1.  Prediction  of  Reliability  and  Maintainability  Function 

REFERENCES: 

a.  System  Reliability  Prediction  by  Function  RADC-TRD- 
63-300  ARINC  Research  Corp.  (May  1963),  (Supplement  1  Mar  1965). 

b.  System  Reliability  Prediction  by  Function  RADC-TDR- 
63-146  Federal  Electric  Corp.  (May  1963). 

c.  Program  to  Establish  Review  Point  Criteria  for  Reliability 
Monitoring,  Hughes  Aircraft  Co.  ,  Publication  52063  (Dec  1964). 

d.  Maintainability  Prediction  by  Unit  Function  RADC-TR-65- 
467  ARINC  Research  Corp.  (Dec  1965). 

e.  Reliability  Prediction  for  Mechanical  and  Electromechanical 
Parts  RADC-TDR-64- 50,  American  Power  Jet  Co. 

f.  Maintainability  Prediction  by  Function  (E.  P.  Simshauser) 

-  Annals  of  Reliability  and  Maintainability,  Volume  5,  Page  421. 

g.  Achieving  System  Effectiveness  through  Reliability  Predic¬ 
tion  (COUTINHO-TIGER)  -  1966  Review  -  Annals  of  Reliability  and 
Maintainability,  Volume  5,  Page  678. 

Reliability  and  maintainability  prediction  is  required  for  equipments 
in  the  early  design  stage  when  information  available  concerns  equipment 
function  rather  than  part  population  and  stress  factors.  A  study  under 
AF  Contract,  AF  30(602)  .  3387  is  developing  techniques  for  predicting 
avionic  equipment  reliability  in  the  early  design  stage.  Reference  (d) 
is  an  interim  report  on  this  contract.  This  study  follows  up  referenced 
earlier  work  by  ARINC,  FED,  and  Hughes  Aircraft.  Little  similar 
work  has  been  done  on  mechanical  equipment. 


178 


Application  of  the  prediction  techniques  to  equipment  which  differs 
in  design  from  those  studied  is  apt  to  be  invalid.  More  work  is  required 
to  devise  prediction  techniques  which  will  have  wide  applicability  in  early 
design  work.  RADC  has  pursued  work  on  Reliability  Prediction  by 
Function  for  the  past  three  years.  A  technique  for  the  ground  environ¬ 
ment  has  been  developed  and  has  shown  high  promise  in  trial  applications. 
It  is  now  being  refined  and  will  be  available  for  general  use  in  the  latter 
part  of  FY  67.  Very  recent  work  which  demonstrated  the  feasibility  of 
reliability  prediction  by  function  in  the  airborne  environment  will  serve 
as  the  basis  of  FY  67  work  in  both  the  airborne  and  space  environments. 
Preliminary  study  of  maintainability  prediction  by  function  has  also 
shown  promise  and  will  be  the  subject  of  further  study  in  FY  67. 

2.  Prediction  of  Reliability  Degradation  Due  to  Storage  and 

Handling 

REFERENCE:  Dormant  Operating  and  Storage  Effects  on  Electronic 
Equipment  and  Part  Reliability  RADC-TR-65- 323. 

Too  often,  there  is  considerable  lapse  of  time  between  Government 
acceptance  and  field  use.  Information  is  needed  on  the  effect  of  storage 
and  handling  on  reliability.  Studies  of  shelf  life  of  components  and 
whole  equipment  is  required. 

RADC  now  has  under  way  a  study  of  the  effects  of  dormancy  and 
storage  on  the  reliability  of  electronic  equipment.  This  study,  which 
is  referenced  above,  is  being  performed  at  the  request  of  Ballistic 
Systems  Division  and  will  be  completed  in  the  first  quarter  of  FY  67. 

3.  Demonstration  of  High  Reliability  Requirements 

High  MTBF  equipment  poses  a  serious  problem  in  demonstra¬ 
tion  on  a  statistically  sound  basis  prior  to  acceptance.  A  combination 
of  limited  testing  and  application  of  analytical  techniques  may  offer  a 
reasonable  compromise.  Study  to  develop,  validate,  and  document 
suitable  techniques  are  necessary. 

4.  Demonstration  of  Reliability  of  "Few  of  a  Kind"  Systems 

Few  of  a  kind  systems  present  a  p- «blem  in  deriving  a  statisti  ¬ 
cally  sound  reliability  quantification.  In  complex  systems,  interdepen¬ 
dence  of  systems  and  subsystems  is  not  readily  determinable.  New 
techniques  are  needed. 

Methods  by  which  preliminary  design  assessment  (including 
apportionments)  can  be  improved,  should  be  found  and  introduced  into 
design  development.  While  current  research  efforts  largely  result  in 
unique  analytic  models  applicable  to  a  single  system,  there  are 
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attempts  under  way  to  generalize  analytic  techniques  for  much  wider 
application.  Further  research  is  needed. 

5.  Nondestructive  Test  Techniques  as  an  Assessment  Tool 
REFERENCES: 

a.  Proceedings.  Tenth  National  Symposium  on  Reliability 
and  Quality  Control,  P.  218(9  Feb  1964). 

b.  "Complementary  Roles  of  Destructive  and  Nondestructive 
Testing  in  Development  Programs" --Nondestructive  Testing  XVII  No.  2 
pp  97-106  (Mar-Apr  1959). 

c.  International  Science  and  Technology  No.  31  ( Jul  1964). 

d.  Proceedings,  Eleventh  National  Symposium  on  Reliability 
and  Quality  Control,  pp  202-222  (Jan  1965). 

e.  Mechanical  Signature  Analysis,  A  New  Tool  for  Product 
Assurance  and  Early  Fault  Detection  (WEICHBRODT)- -Annals  of 
Reliability  and  Maintainability,  Volume  5,  page  569. 

Nondestructive  testing  (NDT)  is  a  source  of  unique  reliability  data 
which  augments  data  from  destructive  and  environmental  testing. 

Nondestructive  test  analysis  is  a  means  of  correlating  development 
and  production  test  data. 

NDT  employing  ultrasonics,  magnetics,  infrared  photography, 
X-ray,  eddy  current  and  other  "senses"  extends  investigation  beyond 
natural  (human)  limits.  Research  is  needed  on  translation  of  non¬ 
destructive  testing  results  into  meaningful  reliability  assessment  data. 

6.  Physics  of  Failure  as  an  Assessment  Tool 

Physics  of  failure  studies  are  being  pursued  o u  many  electronic 
parts  and  integrated  circuits  and  to  a  lesser  extent,  mechanical  devices. 
These  studies  should  be  continued  and  expanded  especially  on  nonelec¬ 
tronic  devices.  Failure  mode  and  failvre  mechanism  studies  are  under 
way  which  provide  a  basis  for  reliability  and  maintainability  assess¬ 
ment  for  mechanical  devices.  M  're  research  is  needed  in  this  area. 

7.  Improved  Maintainability  Prediction 

REFERENCES: 

a.  Maritime  Administration  Maintenance  and  Reliability 
Program --Dunlap  Associates,  Inc.,  MA-3402  4  Vols.  (Mar  1965). 

b.  Status  Reports,  ARINC  Research  Corp.  MDCS  Data  Feed¬ 
back  Analysis  for  the  U.  S.  Naval  Applied  Science  Lab.  --Contract  No. 
N001 40-66 -X01 51. 
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c.  Maintainability  Engineering,  USAMC/Martin-Orlando, 

A  great  need  exists  for  improvements  in  failure  data  collection 
and  analysis  to  enhance  maintainability  prediction  efforts. 

An  accurate  measurement  is  needed  of  time  and  money  expended  on 
maintenance  action  including  measure  of  active  maintenance  time,  the 
level  of  maintenance  action,  all  the  costs  involved,  logistic  support  and 
downtime. 

Example:  A  Maritime  Administration  Study  suggests  that 
"a  complete  expression  of  maintenance  costs  would  have  to 
include  all  categories  of  expense  brought  about  by  the 
potential  and  actual  occurence  of  deterioration  and  failure, 

II 


"Even  a  proportion  of  fuel  consumption  could  be  attributed 
to  the  need  for  transporting:  maintenance  machinery,  equip¬ 
ment,  personnel  and  parts.  " 

Properly  planned  preventive  maintenance  may  reduce  unscheduled 
maintenance.  Excessive  preventive  maintenance  may  increase  expense 
without  producing  offsetting  gains. 

8.  Reliability  and  Maintainability  Prediction  of  Mechanical 
Equipment 

Development  of  procedures  for  predicting  Maintainability  of 
machinery  and  of  whole  systems  is  badly  needed.  Current  and  planned 
RADC  work  in  maintainability  prediction  by  function  was  mentioned 
above.  Other,  directly  applicable,  current  work  includes  a  RADC  study 
of  the  effects  of  maintenance  personnel  on  equipment  downtime  and  a 
Navy  (Marine  Engineering  Lab)  study  on  development  of  reliability  and 
maintainability  functions  for  a  ship  propulsion  system.  More  emphasis 
is  needed  in  this  area. 

9.  Use  of  Quality  Control  Data  for  Assessment 

A  tremendous  volume  of  quality  control  data  is  generated  during 
the  production  phase  which  is  not  generally  used  in  reliability  or  main¬ 
tainability  assessment.  Panel  6  considered  this  data  as  a  potential 
source  of  valuable  information  on  critical  component  failure  rates, 
early  failures  and  burn-in  history.  Such  test  data  should  be  useful  in 
shortening  reliability  test  time.  The  contractors  production  repair 
and  rework  cycle  provides  a  source  for  evaluation  of  equipment 
reliability.  Also,  too  of‘rtr.  problems  in  the  field  on  reliability  can  be 
traced  to  inadequate  quality  control.  Research  into  correlations  between 
acceptance  rates  and  reliability  characteristics  should  be  fruitful. 
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10.  Reliability  Demonstration  Using  Accelerated  Testing 


Studies  have  been  made  on  accelerated  testing  effects  and  their 
correlation  with  reliability  and  maintainability  characteristics.  Con¬ 
tinued  research  is  needed  in  the  following  areas:  design  of  tests, 
analysis  of  test  data,  formulation  of  models  for  prediction,  effects  of 
combined  environments  and  methods  of  applying  such  combined  stresser 
to  provide  optimum  data  within  reasonable  costs.  Furthermore,  in  the 
face  of  newly  developed  products,  such  as  integrated  circuits,  continuous 
research  is  needed  to  study  how  these  stresses  effect  these  new  devices. 

1 1 .  Human  Aspects  of  Maintainability  and  Reliability 
REFERENCES: 

a.  Annals  of  Reliability  and  Maintainability,  Vol  5,  page  112, 
"Armnet  -  A  Quantitative  Approach  to  Evaluation  of  Man  -  Machine 
System  Availability.  " 

b.  Annals  of  Reliability  and  Maintainability,  Vol  5,  page  116, 
"A  Methodology  to  Analyze  and  Evaluate  Critical  Human  Performance.  " 

c.  Annals  of  Reliability  and  Maintainability,  Vol  5,  page  123, 
"Human  Factors  and  Systems  Effectiveness." 

Many  attempts  have  been  made  to  evaluate  the  human  variable  in 
the  reliability  and  maintainability  equation  with  some  success.  The 
literature  leaves  much  to  be  desired  with  the  obvious  conclusion  that 
continued  research  is  needed  into  the  human  factor  effect  on  reliability 
and  maintainability  as  more  sophisticated  equipment  is  fielded. 

12.  Reliability  and  Maintainability  Trade-off  Techniques 

REFERENCES: 

a.  Annals  of  Reliability  and  Maintainability  ,  Vol  5,  page  182, 
"Cost  Effectiveness  Session  -  Introductory  and  closing  Remarks.  " 

b.  Annals  of  Reliability  and  Maintainability,  Vol  5,  page  310, 
"Reliability  and  Maintainability  Cost  Trade-off  via  Dynamic  and  Linear 
Programming.  " 

Reliability  and  maintainability  interests  are  frequently  interrelated 
creating  a  need  for  trade-off  studies.  The  manager  confronted  with  the 
decision  needs  a  visibility  which  is  not  easily  providied.  He  needs  an 
ability  to  judge  the  potential  improvement  of  each  parameter  to  deter¬ 
mine  the  relative  allocation  of  resources.  Research  into  possible 
models  to  express  the  impacts  of  trade-offs  in  a  language  clearly  under¬ 
stood  by  the  responsible  manager  is  needed.  The  literature  takes  full 
cognizance  of  the  dearth  of  information  in  this  area. 


The  above  list  is  not  complete,  but  it  does  include  most  of  the 
urgent  research  needs.  Some  of  these  problems  are  the  subject  of 
intensive  current  research  while  others  have  not  yet  been  seriously 
addressed. 

Research  efforts  are  incomplete  if  the  resulting  techniques  are 
not  validated.  Too  often,  good  research  efforts  fail  to  result  in  useful 
end  products  because  this  vital  step  is  slighted  or  omitted. 


Recommendations 


It  is  recommended: 

1.  That  the  Services  intensify  research  studies  (to  include  tech¬ 
nique  validation)  in  the  areas  listed  above  under  the  guidance  01  the 
Reliability  and  Maintainbility  Steering  Committee.  (See  recommenda¬ 
tion  under  Inter  service  Coordination  page. ) 

2.  That  OSD  establish  a  line  item  in  the  budget  structure  for 
"Research  and  Technique  Development  for  Reliability  and  Maintain¬ 
ability.  " 
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USE  OF  RELIABILITY  AND  MAINTAINABILITY 
ASSESSMENT  RESULTS  BY  MANAGEMENT 


The  purpose  of  a  reliability  and  maintainability  assessment  program 
is  to  provide  meaningful  decision-making  tools  for  efficient  project 
management.  Ideally,  the  decision  maker  should  have  available  to  him 
the  projected  impact  of  available  alternatives  on  reliability  and  main¬ 
tainability  in  a  form  which  will  allow  him  to  trade  off  these  parameters 
against  other  performance  parameters,  cost,  and  schedi-e.  Often  in 
the  past,  reliability  and  maintainability  programs  have  failed  to  provide 
the  decision  maker  with  timely,  useful,  and  adequate  information.  The 
result  is  that  in  far  too  many  cases  reliability  and  maintainability  have 
come  out  on  the  short  end  in  the  decision  making  process.  This  has  not 
been  due  to  a  lack  of  concern  on  the  part  of  management.  Rather,  deci¬ 
sion  makers,  when  weighing  a  subjective  opinion  that  reliability  may  be 
adversely  affected  against  a  quantified  increase  in  performance  or 
decrease  in  immediate  cost,  will  normally  react  in  favor  of  the  certain 
consequence  and  assume  the  reliability  problem  can  be  overcome. 

Panel  6  reviewed  this  last  vital  link  in  the  reliability  and  maintain¬ 
ability  assessment  program  chain  to  determine  what  factors  cause  this 
lack  of  effective  communication  with  management  and  arrived  at  the  fol¬ 
lowing  conclusions: 

1.  Reliability  and  maintainability  assessment  inputs  to  manage¬ 
ment  e  not  made  on  a  consistent  and  continuous  basis  across  the  life 
cycle 

2.  Too  often,  reliability  and  maintainability  assessments  give  a 
description  of  the  existing  situation  but  do  not  provide  a  basis  for  exam¬ 
ining  alternate  courses  cf  action. 

3.  Reliability  and  maintainability  assessments  seldom  provide  a 
means  of  establishing  priorities  for  corrective  action  in  terms  of  a 
meaningful  value  system. 

4.  Reliability  and  maintainability  assessment  results  are  rarely 
in  a  form  which  allows  quantitative  trade-off  with  other  performance 
parameters;  e.  g.  ,  should  additional  development  effort  be  expended  on 
improvement  in  reliability,  or  would  it  be  better  spent  on  accuracy 
improvement? 

Until  the  above  deficiencies  are  corrected,  reliability  and  maintain¬ 
ability  programs  will  largely  remain  outside  the  main  stream  of  manage¬ 
ment.  Far  too  often,  quantified  reliability  and  maintainability 
assessments  are  generated  by  reliability  and  maintainability  personnel 
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for  consumption  by  other  reliability  and  maintainability  personnel  and 
are  regarded  as  interesting  but  by  no  means  vital  by  key  decision  makers. 

Panel  6  concluded  chat  this  problem  cannot  be  solved  by  policy  or 
procedural  changes.  When  the  reliability  and  maintainability  assess¬ 
ment  programs  have  matured  and  can  make  the  proper  information 
available  to  management,  it  will  be  used.  Hence,  corrective  action 
must  take  three  forms:  first,  improvement  of  techniques  to  make 
assessment  results  more  valid;  second,  familiarize  management  with 
specialized  terminology;  and,  finally,  increased  in-house  competence 
in  reliability  and  maintainability  assessment.  These  actions  should 
create  the  proper  environment  for  an  effective  dialog  between  manage¬ 
ment  and  reliability  and  maintainability  personnel. 


Recommendations 


It  is  recommended: 

1.  That  .ne  Services  initiate  '‘cradle  to  grave"  effectiveness 
assessment  programs  to  provide  a  consistent  basis  for  trade-offs 
between  reliability  and  maintainability  and  ether  performance  param¬ 
eters.  (See  recommendation  under  Consistent  Assessment  Criteria  for 
Reliability/ Maintainability.  ) 

2.  That  the  Services  place  high  priority  on  the  establishment  and 
use  of  assessment  techniques  which  establish  meaningful  priorities  for 
corrective  action  and/or  are  specifically  designed  to  aid  the  executive 
in  the  decision-making  process. 

3.  That  the  Services  undertake  continuing  programs  to  train 
management  personnel  at  all  levels  in  reliability  and  maintainability 
terminology  while  simultaneously  reducing  specialized  terminology  to 
the  minimum. 
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INTERSERVICE  COORDINATION 


It  was  the  unanimous  judgment  of  the  members  of  Panel  6  that  con¬ 
tinued  progress  in  reliability  and  maintainability  assessment  can  best 
be  served  by  closer  and  more  formalized  coordination  among  the 
services.  A  tri-service  committee  under  the  chairmanship  of  OSD 
having  advisory  members  from  DSA,  NSA,  NASA,  and  other  interested 
agencies  is  vitally  needed.  This  committee  would  serve  a  dual  purpose 
of  exchanging  information  among  practitioners  of  the  reliability  and 
maintainability  art  and  acting  in  an  advisory  capacity  to  OSD  and  the 
departments. 


Rec  ommendation 

It  is  recommended  that  OSD  form  a  Reliability  and  Maintainability 
Assessment  Steering  Committee  to  exchange  information  and  to  advise 
OSD  and  the  departments  regarding  reliability  and  maintainability 
assessment.  Specific  areas  addressed  by  this  committee  should  include 
the  following: 

1.  Reliability  and  maintainability  technique  research  and  valida¬ 
tion- -determine  needs,  recommend  priorities,  avoid  duplication,  and 
disseminate  results. 

2.  Reliability  and  maintainability  methodology  and  terminology- - 
assure  uniformity,  recommend  changes,  and  disseminate  information. 

3.  System  effectiveness  data  system  design  to  insure  consistency 
and  continuity  throughout  all  system  life  cycle  phases. 
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Appendix  A 


PROPOSED  DEPARTMENT  OF  DEFENSE  INSTRUCTION 

SUBJECT:  Life  Cycle  Assessment  of  Materiel  Effectiveness 

References:  {a)  DoD  Instruction  3200.  6,  "Reporting  of  Research 

and  Engineering  Program  Information.  " 

(b)  DoD  Direction  3200.  9,  "Initiation  of  Engineering  and 
and  Operational  System  Development.  " 

(c)  DoD  Directive  4100.  35,  "Development  for  Integrated 
Logistic  Support  for  Systems  and  Equipment.  " 

(d)  DoD  Directive  4155.  11,  "Improved  Management  for 
Quality  and  Reliability  Assurance  of  Materiel.  " 

I.  PURPOSE 

This  Directive  establishes  DoD  policies  and  objectives  governing 
the  life  cycle  analysis  and  assessment  of  materiel  effectiveness 
to  assure  mission  accomplishment,  assigns  responsibilities  and 
directs  appropriate  actions  for  carrying  out  the  program. 

II.  APPLICABILITY 

The  provisions  of  this  Directive  apply  to  the  Departments  of  the 
Army,  Navy,  and  Air  Force,  and  the  Defense  Supply  Agency. 

HI.  DEFINITIONS 

For  purpose  of  this  Directive,  definition  of  terms  not  defined 
herein  are  in  accordance  with  MIL-STD-280  and  MIL- STD- 72 IB. 

A.  ASSESSMENT- -The  quantitative  determination  of  predicted  or 
observed  performance  of  materiel  against  pre-established 
effectiveness  criteria. 

IV.  CONCEPT  AND  OBJECTIVES 

A.  The  primary  objective  of  life  cycle  assessment  is  to  provide 
management  with  continuous  information  reflecting  reliability 
and  maintainability  status  in  the  context  of  materiel  effective- 
ne:  s.  This  assessment  is  based  on  a  concept  of  establishing 
criteria  for  measurement  of  materiel  effectiveness  at  the  out¬ 
set  of  a  program.  These  criteria  are  then  held  constant  from 


the  initiation  of  engineering  development  through  deployment 
and  subsequent  reprocurements. 

B.  The  overall  objectives  of  the  DoD  Assessment  Program  are: 

1.  To  provide  a  quantitative  measure  of  Materiel 
Effectiveness. 

2.  Provide  consistent  and  compatible  criteria  for  logistic 
support  of  systems  and  equipment. 

3.  Provide  a  means  by  which  the  Departments  can  make 
effective  trade-offs  and  take  timely  action  for 
improvement. 

V.  RESPONSIBILITIES 

The  ASD-(  )*will  provide  overall  policy  guidance  for  the 

systems/equipment  assessment  program  and  shall  monitor  its 
implementation  to  assure  attainment  of  the  objectives  outlined  in 
the  above. 

VI.  MANAGEMENT  ACTIONS 
The  Departments  will: 

A.  Establish  procedures  that  assure  life  cycle  assessment  of 
major  systems  and  equipment. 

B.  Utilize  formal  modeling  and  statistical  methodology  in  the 
assessment  and  prediction  of  systems /equipment  effectiveness. 

C.  Provide  for  the  orderly  transition  of  data  throughout  each 
phase  of  a  systems/equipment  life  cycle. 

D.  Insure  the  inclusion  of  Quantitative  Reliability  and  Maintain¬ 
ability  requirements  in  all  system/equipment  contracts 
including  rebuys. 

E.  Perform  early  assessments  of  fundamental  requirements 
documents  to  determine  feasibility  of  achieving  stated 
reliability  and  maintainability  requirements. 

F.  Maintain  and  insure  adequate,  compatible  data  feedback  to 
support  the  assessment. 

‘The  appropriate  organization  to  be  designated  by  OSD. 
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G.  Establish  procedures  which  provide  for  timely  availability 
and  maximum  utilization  of  assessment  results  at  all  levels 
of  DoD. 

H.  Develop  figures  of  merit  by  which  system/equipments  can  be 
compared  within  common  environments. 

I.  Perform  these  assessments  as  objectively  as  practicable. 
In-house  assessments  should  be  performed  to  the  extent 
possible  within  manpower  restrictions.  If  this  is  not  practical, 
a  "third  party"  contractor  (not  hardware  oriented)  should  be 
used.  If  it  is  necessary  to  have  assessments  performed  by 
prime  contractors  they  will  be  reviewed,  monitored  and 
approved  by  knowledgeable  in-house  personnel. 

VII.  This  Directive  is  effective  immediately. 
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Appendix  B 


PROPOSED  ADDITION  TO  MIL- STD- 78  5 


5.  1 .  4 1  Coordination  of  Test  Requirements  and  Test  Plans.  Reliability- 
data  requirements  shall  be  integrated  into  all  types  of  tests  such  as 
proof  of  design,  breadboard,  prototype,  environmental,  production  and 
acceptance.  In  order  to  achieve  this,  sample  sizes,  data  requirements, 
definitions  of  failure,  and  special  instrumentation  requirements  shall  be 
coordinated  with  those  primarily  responsible  prior  to  the  conduct  of  test. 


'Subsequent  paragraphs  should  be  renumbered. 
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Appendix  C 


RELIABILITY  AND  MAINTAINABILITY  DATA 
STORAGE  AND  RETRIEVAL  SYSTEMS 


Some  of  the  types  of  data  storage  and  retrieval  systems  now  in 
operation  within  DoD  and  NASA  are  briefly  described  in  this  appendix. 


FARADA- -FAilure  RAte  DAta 

Navy  Weapons  Command  established  the  FARADA  program  (for 
Fleet  Ballistic  Missile  Program)  to  collect,  summarize,  analyze, 
compile,  and  distribute  failure  rate  data.  Close  to  one  hundred  organi¬ 
zations  and  facilities  participate  in  this  program.  The  failure  rate 
data  initially  contributed  by  the  participants  was  compiled  as  the 
"Failure  Rate  Data  Handbook,  "  SP  63-470,  which  was  distributed  in 
June  1962;  the  volumes  (2)  are  updated  quarterly. 

Volume  I  presents  tabulated  data  divided  into  the  following  sections: 
Electrical,  Electronic,  Mechanical,  Hydraulic,  Pneumatic,  Prototechnic, 
and  Miscellaneous. 

Volume  II  presents  stress  curve  data  including  source  "Background 
Information"  to  provide  the  required  degree  of  validity  and  applicability 
for  the  Volume  I  data. 

The  tabulated  data  uses  computer  storage,  manipulation,  and 
retrieval  techniques.  FARADA  is  at  present  being  supported  by  the 
three  departments  and  NASA. 


IDEP- -Inter service  Data  Exchange  Program 

This  program,  sponsored  jointly  by  the  three  departments,  provides 
for  the  interchange  of  reliability  test  data  on  missile  and  aerospace 
components  and  parts.  The  charter  of  IDEP,  as  presently  set  up,  allows 
participation  only  by  prime  contractors  (and  major  subcontractors  on 
certain  programs)  engaged  in  ballistic  missile  and  space  hardware  pro¬ 
jects  for:  The  Air  Force  Ballistic  Systems  Division,  Space  Systems 
Division,  The  Army  Missile  Command,  or  the  Navy  Special  Projects 
Office. 


To  utilize  IDEP,  a  contractor  mails  a  summary  sheet  and  two 
copies  of  his  report  to  the  data  distribution  center  of  the  cognizant 
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service.  The  center  microfilms  the  entire  report  and  adds  a  summary, 
based  upon  that  accompanying  the  report  from  the  contractor,  to  a 
microfilm  card.  The  only  alterations  made  in  submitted  material  are 
those  that  may  be  necessary  to  comply  with  standard  formats.  The  film- 
card  combinations  are  then  mailed  to  all  contractors  interested  in  die 
topic  on  a  monthly  basis. 

The  processing  of  the  data  into  historical  data  files  provides  up- 
to-date  reliability  performance  criteria  that  are  especially  useful  when 
the  files  are  subsequently  used  as  the  data  source  for  a  reliability  pre¬ 
diction  and  analysis  program. 


ECRC- -Electronic  Component  Reliability  Center 

The  Electronic  Component  Reliability  Center  (ECRC)  was  intended 
at  Battelle  Memorial  Institute,  Columbus,  Ohio,  on  1  March  1959. 
Beginning  with  eight  industrial  sponsors,  its  membership  has  now  grown 
to  include  Government  agencies.  The  Center  is  supported  through  indi¬ 
vidual  contracts  with  Battelle,  and  are  renewed  each  year. 

The  objects  of  the  ECRC  program  are:  the  extraction  of  reliability 
information  from  sponsor  data,  development  of  techniques  for  reliability 
analysis,  and  assisting  members  in  the  application  of  those  techniques. 

The  ECRC  Data  Center,  therefore,  is  a  clearing  house  for  all 
available  information  about  the  types  and  performance  characteristics 
of  electronic  parts  of  particular  interest  to  its  members.  The  various 
data  outputs  are  as  follows: 

1.  Data  Summaries-  -data  from  test  reports  received  from  the 
members. 

2.  A  Unified  Index  which  lists  all  reports  from  members  with  a 
code  that  indicates  the  location  of  additional  report  information  of 
special  interest. 

3.  Technical  Memoranda- -describe  the  effects  of  particular 
environments  on  various  part  types,  and  also  give  special  listings  of 
pertinent  information. 

4.  Special  Services  which  provide  individual  members  with  answers 
or  information  about  particular  devices. 
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PRINCE--Parts  Reliability  Information  Center 

PRINCE  is  a  specialized  data  center  developed  and  maintained  by 
the  NASA  George  C.  Marshall  Space  Flight  Center  (MSFC).  The 
PRINCE  provides  an  automated  data  storage  and  retrieval  system  that 
contains  technical  information  useful  in  a  parts  reliability  program;  it 
includes  over  20,000  test  reports  on  16,000  individual  parts. 

Maximum  use  is  made  of  other  parts-data  centers  such  as  IDEP 
(Interservice  Data  Exchange  Program)  and  ECRC  (Electronic  Component 
Reliability  Center)  at  Battelle  Memorial  Institute.  In  addition  to  these 
sources,  a  great  deal  of  information  comes  from  MSFC  and  other  NASA 
centers  and  contractors. 


NEDL- -Naval  Electronic  Data  Library 

The  Naval  Electronic  Data  Library  (NEDL)  documents  the  electrical 
and  physical  descriptions  of  all  Naval  electronic  equipments  from  the 
system  level  down  through  the  functional  circuit  level.  This  data  library 
is  being  used  to  provide  detailed  functional  studies  for  each  of  the  systems 
documented,  and  assists  in  determining  the  number  and  types  of  high 
usage  functions  that  are  required  in  each  system.  Included  in  this 
library  are  such  additional  details  as  cost,  reliability,  MTTR,  physical 
size,  and  weight. 

The  control  center  for  this  library  is  located  at  the  Institute  of 
Cooperative  Research,  University  of  Pennsylvania,  Philadelphia, 
Pensylvania,  whose  mission  it  is  to  continually  update  the  library  with 
the  latest  system  developments,  and  to  provide  information  processing 
services  for  its  users. 


Reliability  Central  (Rome  Air  Development  Center) 

The  development  and  establishment  of  a  "Reliability  Central"  at 
Rome  Air  Development  Center  is  now  under  way.  The  Central  will 
serve  as  the  Air  Force  focal  point  for  the  acquisition,  storage,  analysis, 
and  dissemination  of  reliability  information.  Initially,  the  Central's 
operation  will  be  limited  to  electronic  part  types  covered  by  Federal 
Stock  Class  59.  Eventually,  the  operation  will  encompass  semicon¬ 
ductor  integrated  circuits,  mechanical  and  electromechanical  parts, 
equipment,  subsystems  and  systems.  (Prior  to  the  establishment  of 
a  full-scale  Central,  a  test  operation  on  transistors  and  diodes  will  be 
developed  and  implemented  to  demonstrate  the  feasibility  and  potential 
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of  a  full-scale  system. . . . )  The  expansion  of  the  Central's  file  to 
incorporate  reliability  information  on  equipment,  systems,  and  mechani¬ 
cal  parts  and  assemblies  will  be  accomplished  in  an  orderly  manner. 


FMSAEG--Fleet  Missile  System  Analysis  and  Evaluation  Group 

FMSAEG  has  an  integrated  program  for  the  collection,  processing, 
and  analysis  of  reliability,  operability,  and  component  part-failure 
data  for  surface-missile  systems.  While  the  major  portion  of  the  data 
collection  function  will  be  replaced  by  the  MDC  system,  the  FMSAEG 
processing  and  analysis  will  contm-ue.  During  the  transition  period, 
both  FMSAEG  and  MDC  systems  will  be  used. 

FMSAEG  has  the  following  three  principal  areas  of  analysis: 

1.  Surface-missile  flight  analysis- -firing  reports,  telemetry 
records,  and  flight- test  scoring. 

2.  Surface -missile  test  equipment  and  missile  checkout  experi- 
ence--failure  rate  tabulations  of  missile  modules  and  checkout 
equipment. 

3.  Analysis  and  summarization  of  certain  fleet  equipment  data  to 
yield  measures  of  average  up  time  in  various  operating  states.  This 
type  of  analysis  produces  a  gross  measure  of  readiness. 


MDC --Maintenance  Data  Collection  System 

MDC  has  been  designed  as  a  uniform,  Navy-wide  failure  data 
reporting  system.  Data  collection,  purification,  and  processing  MDC 
(implemented  in  1965)  is  performed  by  the  Maintenance  Support  Office 
(MSO),  Mechanic sburg,  Pennsylvania. 

MDC  data  outputs  mainly  consist  of  general  and  summary  data  runs; 
more  elaborate  data  runs  and  analyses  will  be  performed  by  the  inter¬ 
ested  bureaus,  laboratories;  and  other  agencies  which  utilize  the  basic 
data.  The  prime  BUWEPS  Agency  operated  by  the  Armament  Mainte¬ 
nance  Management  Information  Center,  Naval  Weapon  Station,  Concord, 
California,  will  be  responsible  for  ship  data.  Air  data  will  be  handled 
by  the  Naval  Air  Technical  Services  Facility,  Philadelphia,  Pennsylvania. 
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66-1  MAINTENANCE  DATA  COLLECTION  SYSTEM 


The  Air  Force  Maintenance  Data  Collection  System  as  described  in 
AFM  66-1  provides  the  necessary  information  with  which  to  manage  base- 
level  maintenance.  Additionally,  this  information  is  being  used  in 
reliability,  maintainability,  spares  provisioning,  etc.,  on  operational 
system  or  equipments.  Data  collection  is  at  the  base  (flight  line  and 
shops)  level.  It  is  forwarded  to  HQ,  AFLC,  Wright-Patterson  Air  Force- 
Base,  Ohio,  Where  it  is  processed. 

AFM  66-1  data  outputs  are  available  to  interested  commands  and 
contractors.  It  consists  of  general  and  summary  products  with  special 
purpose  products  available  on  request.  Raw  data,  processed  to  magnetic 
tape,  is  made  available  to  subcommands  of  AFLC  for  detail  reliability 
and  maintainability  analysis. 


THE  ARMY  EQUIPMENT  REPORTING  SYSTEM 


TAERS 

Maintenance  data  are  collected  from  users  of  Army  equipment, 
summarized  and  analyzed.  Data  elements  are  identified  in  computerized 
form  permitting  rapid  feedback  tc  us:  rs  of  these  data.  Additionally, 
TAERS  provides  for  the  submission  of  Equipment  Improvement  Reports 
which  ar  .  furnished  to  Army  design  and  development  activities.  The 
TAERS  Data  Central  is  located  at  Lexington,  Ky.  ,  under  the  direction 
of  the  Army  Maintenance  Board.  Each  commodity  command  of  the 
Army  Materiel  Command  has  computer  facilities  to  handle  the  tapes 
furnished  by  the  Data  Central  and  print  out  pertinent  information  needed 
to  support  the  assigned  maintenance  mission. 


ADRES--Army  Data  Retrieval  Engineering  System 

Engineering  data,  such  as  standard  ami  commercial  drawings, 
specifications,  standards,  and  package  data  sheets,  can  be  automatically 
retrieved  from  this  data  bank.  It  contains  over  200,000  documents  and 
is  drawn  on  by  15  industry  contractors  and  60  in-house  stations.  The 
data  comes  from  in-house  Army  sources  and  is  stored  in  microfilm 
cartridges.  The  output  forms  are  either  visual  displays  or  hard  copies. 

The  data  is  obtainable  in  approximately  one  \  eek.  if  it  is  in-house, 
or  in  three  to  four  weeks  if  it  must  be  accumulated. 


195 


MEDAL  -  Micro  Miniaturized  Engineering  Data  for  Automated 

Logistics 

This  data  bank  contains  engineering  drawings  of  interest  to  the 
U.  S.  Air  Force.  There  are  over  6  million  drawings  on  file. 

The  bank  is  used  primarily  by  Air  Force  in-house  agencies,  and 
it  processes  more  than  1,  000,  000  data  requests  per  month.  The  data 
is  obtained  from  Air  Force  contractors  and  is  stored  in  aperture  cards. 
The  information  output  of  this  system  is  in  the  form  of  aperture  cards 
and  hard  copies. 


EDS- 0009  -  Engineering  Data  System- 0009 

This  data  system  is  a  breadboard  model  of  an  automatic  storage 
and  retrieval  system  for  engineering  information.  It  contains  configura¬ 
tion,  test,  maintainability,  and  reliability  data,  and  package  data  sheets 
for  standardization.  At  present,  the  data  bank  has  engineering  infor¬ 
mation  on  15,  000  capacitors,  15,  000  relays,  8,  000  metals,  and  1,000 
other  materials.  The  data  has  been  accumulated  from  Federal  and 
international  agencies,  including  NATO  industry.  These  data  suppliers 
are  also  the  primary  customers.  Information  is  stored  on  microfilm 
cartridges  and  magnetic  tape.  Outputs  are  in  the  form  of  hard  copies, 
facsimiles,  and  visual  displays. 
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REPORT  OF  PANEL  7 


TITLE:  Personnel  and  Training 


OBJECTIVE: 


To  recommend  plicies,  procedures  and  actions  essential 
to  insuring  qualified  personnel  for  the  management  and 
implementation  of  quality  and  reliability  operations. 


TOPICS  DISCUSSED: 


1.  The  Diagnosis  of  the  Personnel  and  Training  Problem  as 
it  Relates  to  Quality  and  Reliability  in  the  DoD. 

2.  A  Total  Quality  and  Reliability  Career  Program. 


3.  Acquisition.  Development  and  Retention  of  Quality  and 
Reliability  Skills. 
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Mr.  Lou  Korn,  SUPSHIPS  Groton  Conn. 
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THE  DIAGNOSIS  OF  THE  PERSONNEL  AND  TRAINING  PROBLEM 
AS  IT  RELATES  TO  QUALITY  AND  RELIABILITY  IN  DoD 


There  are  approximately  15,000  personnel  in  the  DoD  currently 
engaged  in  quality  and  reliability  assurance  activities.  These  people 
are  professionals  and  sub  professionals  employed  throughout  the  logis¬ 
tics  spectrum  from  research  through  standardization  and  maintenance. 
The  bulk  of  these  people  are  engaged  in  the  procurement  segment  of 
the  spectrum  where  inspection  and  quality  assurance  have  traditionally 
been  utilized.  In  spite  of  this  latter  fact,  quality  audits  by  the  military 
services  of  the  material  in  their  Depots  have  indicated  a  distressingly 
high  percentage  of  non- conforming  material. 

In  addition,  personnel  engaged  in  procurement  quality  assurance 
continually  complain  of  weak  or  missing  quality  assurance  criteria  and 
requirements  in  contracts  and  weak  or  missing  quality  assurance 
elements  in  specification.  These  latter  facts  are  testimony  to  the  inef¬ 
fective  performance  of  quality  assurance  personnel  in  the  pre¬ 
procurement  segment  of  the  logistics  sequence  such  as  research, 
development  and  engineering. 

Similar  effects  of  non-performance  or  weak  performance  can  be 
assembled  concerning  the  segments  of  the  logistics  sequence  beyond 
procurement  such  as  storage,  maintenance  and  rework.  In  effect,  the 
above  constitutes  an  indictment  of  the  management  and  application  of 
these  15,  000  personnel  in  DoD.  In  fact,  it  must  be  admitted  that  we  do 
not  seem  to  be  efficiently  and  effectively  utilizing  and  managing  these 
skills  which  we  must  admit  are  extremely  important  to  the  delivery  of 
the  necessary  quality  and  reliability  to  the  military  user. 

When  Panel  7  assembled  to  discuss  the  problem  and  make  recom¬ 
mendations  concerning  improved  career  management  and  training  of 
its  personnel  engaged  in  quality  and  reliability  assurance,  it  found 
itself  unable  to  come  to  grips  with  the  problem.  Discussed  were  known 
weaknesses  in  current  training  programs  such  as  the  apparent  training 
for  training's  sake,  but  it  was  recognized  that  this  was  not  the  basic 
problem,  but  merely  the  symptom  of  a  larger  problem. 

It  soon  became  apparent  that  the  larger  problem  was  the  fact  that 
the  important  expensive  quality  and  reliability  assurance  program  did 
not  exist  as  a  program.  Personnel  engaged  in  the  activities  include 
engineers,  statisticians,  quality  managers,  specialized  technical  skills, 
specialized  commodity  skills,  all  held  together  in  loose-knit  arrange¬ 
ment  by  a  thread  of  quality  assurance  techniques. 
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Problems  were  found  in  the  incorporation  of  the  special  profes¬ 
sional  skills  of  engineering,  science  and  mathematics  into  the  general 
structure  of  quality  and  reliability  because  of  the  lack  of  a  professional 
image.  Many  "spot  management"  programs  and  other  activities  were 
generated  in  reaction  to  serious  but  limited  problems.  We  found  these 
good,  little,  quality  and  reliability  management  programs  working  with 
varying  degrees  of  integration  within  services,  very  little  integration 
between  services,  and  no  major  discoverable  management  guidance 
from  OSD.  Two  points  seemed  to  be  the  heart  of  the  problem,  these 
being  no  quality  and  reliability  assurance  program  identification  and 
no  overall  management  or  recognition. 

Examples  of  good  but  relatively  unintegrated  spot  management 
programs  are  the  special  treatments  found  necessary  by  Admiral 
Rickover  to  achieve  the  ends  he  desired  and  the  special  treatment  that 
NASA  receives  through  specific  direction  by  agreement  with  the 
Secretary  of  Defense.  There  are  many  lesser  examples  throughout 
the  DoD  structure. 

It  was  deduced  that  one  of  the  problems  of  acquisition  and  reten¬ 
tion  of  professional  skills  today  appeared  to  be  the  inability  to  inte¬ 
grate  these  skills  and  professions  effectively  into  one  segment  of  the 
logistics  spectrum  with  almost  no  ability  to  apply  them  on  a  balanced 
basis  in  the  several  segments  where  they  can  be  most  effectively  uti¬ 
lized. 

For  example,  it  is  conjectured  that  more  application  of  quality  and 
reliability  assurance  skills  in  pre-procurement  activities  would  permit 
the  use  of  fewer  and  lower  skills  in  procurement  and  the  ensuing  logis¬ 
tics  segments.  Too  often,  today,  we  are  using  high  skills  in  procure¬ 
ment  quality  and  reliability  assurance  in  order  to  detect  weaknesses 
which  should  have  been  prevented  during  earlier  activities. 

This  important  consideration  cannot  happen  effectively  today 
bees  ase  the  pre-procurement  segment  falls  under  another  element  of 
DoD  cognizance. 


Summary 

The  diagnosis  of  the  Panel  is  that  there  is  no  quality  and  reliability 
program  as  such  and  as  a  result  we  are  dissipating  important  talents 
and  in  effect,  not  managing  the  application  of  these  important  skills. 
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A  TOTAL  QUALITY  AND  RELIABILITY  CAREER  PROGRAM 


There  is  a  lack  of  clearly  defined  and  integraded  quality  and  reli¬ 
ability  career  development  programs  within  the  DoD.  A  series  of 
interfacing,  carefully  structured  programs  designed  to  provide  for 
continuity  of  purpose  for  total  product  effectiveness  is  required  as 
follows: 

1.  Functional  areas  need  to  be  identified  and  relevant  skill  steps 
and  levels  established. 

2.  There  exists  among  project  and  program  managers  an  indif¬ 
ference  to  quality  and  reliability  which  must  be  overcome. 

3.  Quality  and  reliability  milestones  must  be  incorporated  into 
formal  systems  of  management  control. 

4.  There  exists  an  inequality,  both  in  organizational  placement 
and  salary,  between  quality  and  reliability  and  other  major  programs 
which  must  be  reconciled. 

5.  Developmental  guidelines  to  assist  individulas  toward  a  variety 
of  required  combinations  of  disciplines,  management  and  technical 
areas  must  be  prepared. 

6.  There  is  a  critical  shortage  of  qualified  personnel  due  to  such 
factors  as  expansion,  changing  technology,  attrition  and  the  fact  that 
our  educational  institutions  and  service  schools  are  not  geared  to  meet 
our  requirements. 

7.  Currently,  limited  ceilings  and  operational  billets  preclude 
long  range  career  development. 

No  question  has  aroused  more  interest  and  discussion  that  that  of: 
What  would  be  the  appropriate  job  series  of  DoD  civilians  engaged  in 
quality  and  reliability  work?  The  concept  of  a  single  series  was  ana¬ 
lyzed  and  even  though  this  concept  does  have  merit,  it  was  concluded 
that  a  single  series  would  be  restrictive  to  a  point  of  limiting  the  ability 
to  recruit,  train  and  retain  the  types  of  skills  required  to  perform  the 
man/  and  varied  functions  of  product  effectiveness. 

The  four  job  series  presently  used  are:  1900,  1500,  1300  and  800. 
The  preponderance  of  people  in  product  effectiveness  activities  are  in 
the  1900  series.  This  series  initially  was  established  to  accommodate 
traditional  quality  control.  Today,  it  is  being  given  much  broader 
application.  Many  of  the  people  in  management  positions  in  quality  and 
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reliability  are  in  the  800  series  and  this  trend  is  increasing.  This 
often  causes  a  very  capable  employee  in  either  the  1300,  1500,  or 
1900  series  to  find  that  their  career  is  blocked. 

It  is  proposed  that  in  addition  to  the  four  series  that  are  currently 
.  used,  the  340  series  be  used  to  describe  the  functions  of  the  quality  and 
reliability  manager.  Employing  this  series  would  bridge  the  gap  between 
the  professional  and  non-professional  groups  and  also  would  have  the 
very  desirable  advantage  of  affording  all  capable  individuals  an  equal 
opportunity  for  top  management  positions.  Predominantly,  our  800, 

1300  and  1500  peopl  would  contribute  to  complex  system  planning  and 
product  design  during  the  conceptual  and  definition  phase.  Our  1900 
people  predominantly  would  contribute  to  the  assessment  function  so 
vital  to  production,  maintenance  and  field  operations. 

DoD  liaison  with  various  educational  institutions  should  provide 
curricula  guidelines  for  the  preparation  of  graduates  to  enter  the  pro¬ 
duct  effectiveness  program.  This  liaison  to  include  coordination  and 
establishment  of  academic  minimums  as  pre-requisites  to  entering  the 
product  effectiveness  career  field,  a  program  of  education  for  the 
educators  in  the  area  of  quality  and  reliability,  a  coordinated  program 
with  the  professional  and  indxistrial  community,  and  an  evaluation 
medium  whereby  requirements  data  feedback  could  be  obtained  for 
appraisal  and  improvement  of  the  academic  environment. 

Quality  and  reliability  programs  of  a  career  nature  suitable  to  all 
DoD  and  iTASA  personnel  exist  at  the  USAMETA,  The  Air  Force  School 
of  Systems  and  Logistics  and  to  some  degree  at  the  USALMC.  At  the 
present  time,  the  Air  Force  Institute  of  Technology  18  month  program 
leading  to  a  Master  of  Science  Degree  in  Reliability  Engineering  is  the 
only  course  of  its  type  available.  The  shortage  of  persons  with  spe¬ 
cialized  knowledge  in  the  quality  and  reliability  fields  will  not  be 
relieved  through  forthcoming  college  graduates  since  only  a  very  few 
educational  institutions  provide  training  specialising  in  quality  and 
reliability  engineering  as  a  part  of  their  curricula.  Quality  and 
Reliability  Engineers  are  for  the  most  part,  self  taught;  in  fact,  few 
university  professors  arc  qualified  to  teach  reliability,  hence  e  need 
for  educating  educators. 

Of.  vital  importance  is  that  part  of  the  career  pattern  which  includes 
cross-fertilization.  Specifically,  since  the  Product  Effectiveness  Pro¬ 
gram  spans  the  entire  life  cycle,  it  is  vital  that  quality  and  reliability 
managers  have  experience  in  the  research  and  development,  procure¬ 
ment  and  maintenance  program  phase.  In  addition  to  in-house  cross¬ 
fertilization,  tours  of  duty  in  industrial  organizations  should  be  arranged 
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and  pursued.  Conversely,  it  is  believed  desirable  that  industrialists 
with  whom  we  do  contract  business  be  brought  into  the  military  environ¬ 
ment  as  a  part  of  our  cross-fertilization  effort. 

Establishment  of  a  professional  requirements  board  is  a  must;  this 
board  to  be  composed  jointly  of  military  and  civilian  personnel  con¬ 
cerned  with  professionalism  requirements  at  each  level  of  all  phases 
of  the  career  program.  This  would  provide  a  basis  for  a  sound  train¬ 
ing  curricula  as  well  as  standardized  qualification  and  selection  criteria. 
This  is  consonant  with  the  efforts  of  many  of  our  technical  societies 
moving  toward  a  professional  certification  program  in  the  field. 

Today,  in  our  service  schools,  there  are  many  splinter  courses  of 
instruction  given  for  reliability  and  quality.  There  is  not,  at  this  time, 
a  single  school  that  teaches  an  integrated  course  covering  all  facets  of 
the  Product  Effectiveness  program.  It  is  highly  desirable  that  a  survey 
be  conducted  of  the  courses  available  today  with  an  objective  of  con¬ 
solidating  where  practical  and  eliminating  duplication. 


Recommendations 


It  is  recommended: 

1.  .  That  the  OSD  recognize  the  need  for  and  accept  the  responsi¬ 
bility  for  directing  the  establishment  of  quality  and  reliability  assurance 
as  a  program  with  identification  and  entity  applicable  across  the  entire 
logistics  spectrum,  as  well  as  establishing  a  structure  for  its  effective 
focal  management. 

2.  That  a  study  be  directed  which  will  concisely  identify  and 
describe  the  total  product  effectiveness  program,  the  role  it  must  play 
throughout  the  spectrum,  as  well  as  the  several  professional  and  sub¬ 
professional  skills  required.  This  study,  in  addition  to  necessary 
identification,  should  also  result  in  recommended  DoD  Directive  mate¬ 
rial  and  appropriate  modifications  to  existing  directive  material  (ASPR, 
DoD  Directives,  etc. ). 

3.  That  a  career  program  structure  be  developed  which  will 
incorporate  as  part  of  its  structure  the  several  professions  and  skills, 
facilitate  their  proper  application  and  permit  movement  and  growth  of 
military  and  civilian  personnel  horizontally  and  vertically  in  the  career 
ladders  necessary  to  attraction  and  retention  of  the  calibre  of  personnel 
necessary  would  result.  As  a  part  of  the  career  program,  mandatory 
assignment  rotation  both  geographically  and  in  specialized  fields  would 
help  assure  that  a  sufficient  number  of  people  with  board  experience 
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could  rise  to  the  levels  of  mid  and  top  management  necessary  to  the 
broad  guage  concepts  of  the  program.  In  the  development  of  a  career 
program  for  product  effectiveness  personnel,  provisions  be  made  for 
the  transition  from  the  800,  1300,  1500  or  1900  series  into  the  340  or 
other  appropriate  series,  in  order  that  any  member  of  the  product 
effectiveness  team  has  a  defined  career  progression  to  top  manage¬ 
ment  positions. 

4.  That  increased  liaison  be  maintained  with  educational  insti¬ 
tutions  for  the  purpose  of  developing  prerequisite  material  necessary 
to  preparing  college  graduates  for  entry  into  the  product  effectiveness 
career  field. 

5.  That  increased  educational  opportunity  for  selected  personnel 
to  include  such  things  as  scholarships  and  grants  for  advanced  study, 
tuitional  assistance,  job  absence  permissiveness  for  apprentices, 
upgrade  and  graduate  work  for  journeyman  and  comprehensive,  self- 
develop:.ient  program  for  enrollees  in  the  career  field  be  provided. 

6.  That  in-service  and  specialized  training  for  career  progression 
be  made  mandatory  and  be  incorporated.  Here  particular  emphasis 
would  be  placed  on  quality  and  reliability  technical  know-how,  enhancing 
of  the  management  ability  of  the  ind  ddual  exposure  to  peripheral  sys¬ 
tems  such  as  pricing,  contracting  It  al  aspects,  work  measurement, 
budgeting,  s/stem  analysis,  etc, ,  ai  where  applicable,  comprehensive 
structured  on-the-job  training. 

7.  That  specific  DoD-wide  training  agreements  be  developed  to 
attract  suitable  college  talent  and  retain  them  in  the  quality  and  reli¬ 
ability  management  function  by  providing  them  with  an  organized  plan 

of  training  and  development  which  would  include:  training  given  through¬ 
out  the  career  cycle  which  addresses  itself  to  specific  courses  in 
technology,  techniques  and  methodology  contingem  upon  the  immediate 
assignment  of  the  individual  and  his  long  range  objectives. 


ACQUISITION,  DEVELOPMENT  AND  RETENTION  OF  QUALITY 

AND  RELIABILITY  SKILLS 


Approximately  three  fourths  of  the  personnel  engaged  in  product 
effectiveness  activities  will  likely  be  lest  through  attrition  within  the 
next  ten  years.  Sources  of  input  to  replace  these  losses  are  limited 
especially  in  the  engineering  and  management  field.  There  is  a  lack 
of  functional  definition  within  the  career  field  and  an  absence  of  a 
clearly  defined  career  development  program  attractive  to  high  potential 
input.  In  addition,  the  Federal  Government  is  not  competitive  mone¬ 
tarily  in  the  scientific  and  professional  career  fields  insofar  as  the 
recruitment  of  the  young  college  graduate /potential  manager  is  con¬ 
cerned. 

As  a  result  of  the  careful  definition  of  the  role  of  quality  and  reli¬ 
ability,  it  will  be  necessary  to  investigate  the  current  status  of  skill 
availability  and  compare  this  with  the  requirements  as  defined.  Voids 
or  overages  discovered  car.  then  be  resolved.  Positive  steps  can  be 
taken  to  fill  the  voids  through  such  means  as  retraining  personnel  found 
to  be  in  oversupply,  selective  hiring,  cross  training  of  individuals 
already  engaged  in  product  effectiveness  activities,  or  through  the 
assignment  of  military  personnel  with  appropriate  skills. 

Institutional  education,  such  as  colleges  and  universities  lags  tome 
ten  years  behind  demands  of  present-day  technology  insofar  ?.s  product 
effectiveness  is  concerned.  A  study  conducted  by  Catherine  Hock, 
entitled  "Reliability  Engineering  Education  at  Colleges  and  Universities",1 
along  with  a  survey  made  by  Dr.  Dimitri  Kececioglu  and  Mr.  Joe 
McKinley  of  che  Uriiversity  of  Arizona  entitled,  "Reliability  Engineering 
Education  Activities  in  the  United  States  and  Overseas",  2  bears  this  out. 
For  example,  the  Hock  study  reveals  that  approximately  205  accredited 
colleges  and  universities  give  degrees  in  engineering.  Of  these,  38 
give  one  or  more  courses  covering  seme  phase  of  reliability.  However, 
it  was  of  particular  concern  to  note  that  of  the  92%  of  the  engineering 
schools  offering  statistics,  only  2%  or  four  (4)  of  the  universities 
require  it  for  an  engineering  degree.  It  was  also  revealed  that  almost 
all  reliability  courses  were,  in  reality,  only  courses  in  statistics.  This 
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"Reliability  Engineering  Education  at  Colleges  and  Universities", 
Catherine  Hock,  Office,  Manned  Space  Flight,  NASA,  Washington,  D.  C. 

2 "Reliability  Engineering  Education  Activities  in  the  United  States 
and  Overseas",  Dr.  Dimitri  Kececioglu,  Prof.  Aerospace  and  Mechan¬ 
ical  Engineering  and  Joe  McKinley,  Graduate  Associate  in  Research 
and  Teaching,  The  University  of  Arizona. 
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was  true  in  both  the  graduate  and  undergraduate  programs.  The 
University  of  Arizona  study  shows  that  in  addition  to  the  shortcomings 
of  the  engineering  curricula,  there  is  a  lack  of  coordinated  short  courses, 
symposia  and  conferences  of  a  general  interest  nature.  For  the  most 
part,  they  are  highly  specialized  for  a  particular  weapon  system  or 
piece  of  hardware. 

The  few  quality  and  reliability  courses  tailored  to  appeal  to  the 
Defense  establishment,  such  as  the  18  month  program  in  Reliability 
Engineering  conducted  by  the  Force  Institute  of  Technology  leading 
to  a  Master's  Degree  in  Reliability  Engineering  and  the  variety  of  joint 
DoD  short  courses  in  Quality  and  Reliability  presented  by  the  U.  S. 

Army  Management  Engineering  Training  Agency,  the  Air  Force  School 
of  Systems  and  Logistics  and  the  Army  Logistics  Management  Center, 
do  not  satisfy  all  of  the  existing  need,  although  in  many  cases  classes 
go  unfilled.  Again,  the  fact  that  all  of  the  needs  are  not  met  is  due  in 
part  to  the  fact  that  many  of  the  needs  are  not  known  or  defined. 

It  is  obvious,  that  until  such  time  as  university  curricula  frilly  pre¬ 
pare  graduates  for  the  job  of  product  effectiveness,  that  a  continuing 
need  will  exist  for  in-house  career  development  and  training.  A  second 
need  in  this  regard  is  the  requirement  to  train  educators  to  educate. 

At  the  present  time,  the  only  such  program  known  is  one  at  the  Univer- 
sith  of  Arizona  conducted  under  the  sponsorship  of  the  National  Science 
Foundation,  which  is  designed  exclusively  to  train  university  professors 
in  the  tools,  techniques  and  methods  of  presentation  of  the  concepts  of 
quality  and  reliability  in  order  that  they  might  more  effectively  incor¬ 
porate  these  ideas  into  their  own  curricula. 

Another  finding  as  a  result  of  panel  investigation  showed  that  not 
only  are  a  majority  of  individuals  considered  knowledgeable  and  experi¬ 
enced  in  product  effectiveness  about  to  retire,  but  that  many  of  them 
are  assigned  to  functions  which  do  not  adequately  utilize  this  presently 
available  knowledge  and  experience. 

In  order  to  assure  fully  staffed  positions  requiring  product  effec¬ 
tiveness  skills,  it  is  necessary  to  acquire  "new  blood"  immediately; 
but  in  order  to  do  this,  the  positions  must  be  made  attractive  both 
monetarily  and  from  a  career  development  viewpoint  The  functions 
to  be  performed  must  be  clearly  defined  in  order  to  retain  the  indi¬ 
viduals;  and  a  program,  both  formal  and  informal,  mu^t  be  establish'  •. 
whereby  the  individual  is  able  to  constantly  grow. 

An  intensive  campaign  is  required  to  educate  industry  and  educa¬ 
tional  institutions  to  understand  the  important  positive  role  that  product 
effectiveness  plays  in  producing  a  trouble  free  product. 
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Recommendations 


It  is  recommended: 

1.  That  OSD  designate  an  Executive  Agent  within  DoD  to: 

a.  Provide  a  focal  point  for  the  research,  development  and 
monitorship  of  programs  of  instruction  in  the  quality  and  reliability 
area. 

b.  Analyze  the  capability  of  existing  quality  and  reliability 
skills  in  relation  to  current  occupational  requirements  and  qualification 
standards. 

c.  Designate  a  task  group  composed  of  representatives  from 
the  military  services  and  the  Defense  Supply  Agency  to  struc  c  *e  a 
career  development  pr jgr,  n  for  product  effectiveness;  engineers, 
technicians  and  managers. 

d.  E;:amine  existing  programs  of  instruction  in  service 
schools,  academic  and  industrial  facilities  to  establish  comprehensive 
plans  for  the  systematic  development  of  all  product  effectiveness  person¬ 
nel. 


e.  Since  individuals  associated  with  product  effectiveness  r 
should  be  experienced  across  the  entire  logistics  spectrum. 

£.  Encourage  greater  joint  use  of  DoD  Service  schools. 

g.  Extend  blocks  of  instruction  in  quality  and  reliability  into 
the  Senior  Level  Service  Schools  and  Staff  Colleges,  ICAF,  War 
College,  etc.  to  give  management  appreciation  of  the  quality  and  reli¬ 
ability  role  in  DoD  structure. 

h.  Establish  graduate  level  programs  of  instruction  which 
will  permit  the  training  of  professional  and  engineering  personnel  in 
specialization  of  quality  and  reliability  fields. 

i.  Modify  and  extend  the  technical  short  courses  in  the  quality 
and  reliability  field  to  provide  across  the  board  training  of  technical 
and  managerial  personnel  in  Product  Effectiveness. 

2.  That  Military  personnel  who  have  received  and  are  receiving 
specialized  training  in  product  disciplines  be  identified  through  some 
designator  in  order  that  they  might  provide  a  reservoir  for  future 
assignments  which  could  take  advantage  of  their  experience. 
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REPORT  OF  PANEL  8 


T IT LE :  Metrology  and  Calibration  in  Quality  and  Reliability  Operations 
OBJECTIVE: 

To  recommend  action  required  for  the  effective  integration 

and  utilization  of  metrology  and  calibration  in  quality  and 

reliability  operations. 

TOPICS  DISCUSSED: 

1.  Identification  of  New  Measurement  and  Calibration 
Requirements 

2.  Selection  of  Proper  Test  and  Inspection  Equipment  and 
Measurement  Processes 

3.  Promulgation  of  Uniform  Specifications  Controlling 
Contractors'  Calibration  Systems 

4.  Provision  of  Metrology  Support  to  Contract  Administration 
Personnel 

5.  Establishment  of  a  Central  Point  of  Contact  within  Office 
Secretary  of  Defense  Responsible  for  Overall  Policy  for 
Metrology  and  Calibration  and  for  Coordination  with  Other 
Government  Agencies 
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PARTICIPANTS 


Chairman 


Mr.  Jerry  L.  Hayes,  Navy  Metrology  Engineering 
Center 


Panelists 


Lt.  Col.  Gail  D.  Conlee,  USAF,  HQS  Air  Force 


Mr.  Ray  Y.  Bailey,  Newark  Air  Force  Station, 
Ohio 

Mr.  L.  Weissenberger ,  FAA  Depot,  Oklahoma 
City 

Captain  R.  H.  Gillock,  USN,  Fleet  Readiness  Rep, 
Atlantic 

Mr.  Ernie  D.  Stefani,  DCASR,  Detroit 

Mr.  Richard  J.  Bedford,  HQS,  DCASR 

Major  William  A.  Porter,  HQS,  NASA 

Mr.  David  M.  Herzmark,  White  Sands  Missile 
Range 

Mr.  Melvin  Fruechtenicht,  Army  Metrology  and 
Calibration  Center 

Mr.  J.  M.  Cameron,  HQS  NBS 

Dr.  George  E.  Schafer,  NBS,  Boulder 


Recorder 


Mr  David  W.  Worden,  Navy  Metrology  Engineering 
Center 


Monitor 


Mr.  S.  G.  Hamner,  HQS  Naval  Material  Command 
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IDENTIFICATION  OF  NEW  MEASUREMENT 
AND  CALIBRATION  REQUIREMENTS 


Identification  of  new  systems  measurement  requirements  which  are 
beyond  the  current  state-of-the-art  must  be  accomplished  as  early  as 
possible  in  the  materiel  life  cycle,  preferably  in  the  concept  stage,  to 
permit  maximum  lead  time  for  development  of  supporting  standards  and 
instrumentation.  Normally  the  systems  development  contractor  is  in 
the  best  position  to  identify  and  analyze  such  requirements. 

The  services  and  NASA  have  used  several  approaches  to  the  problem 
of  early  identification  of  new  measurement  requirements  with  varying 
degrees  of  success.  Documents  such  as  MIL-Q-9858A  (DoD),  Quality 
Program  Requirements;  MIL-D-9412D  (USAF),  Data  for  Aerospace 
Ground  Equipment;  and  MIL-Q-21549B  (WEP),  Product  Quality  Program 
Requirements  for  Fleet  Ballistic  Missile  Weapon  System  Contractors, 
contain  general  requirements.  More  specific  requirements  have  been 
outlined  in  special  contract  provisions;  however,  there  is  no  uniform 
method  for  submission  of  such  data  prescribed  by  directive  or  ASPR. 

The  nature  of. system  development  is  such  that  priority  of  engineer¬ 
ing  effort  is  normally  directed  at  development  of  operational  hardware. 
Consideration  for  reproducible  measurements  that  will  be  required  to 
support  development  and  testing  is  often  overlooked  until  a  crisis  arises 
in  connection  with  compatibility  of  test  data. 

The  forecast  of  such  measurement  requirements  may  require 
research  or  development  action  by  the  National  Bureau  of  Standards 
to  establish  an  appropriate  national  standard.  Also,  it  will  afford  time 
to  develop  test  equipment  to  measure  the  necessary  parameters  on  a 
production  basis,  which  is  necessary  to  the  successful  production  of 
state-of-the-art  components.  A  hasty  selection  of  inappropriate  equip¬ 
ment  could  thereby  be  avoided. 

A  complete  review  of  the  specifications  and  contract  clauses  now 
included  in  systems  development  contracts  within  DoD  is  necessary  to 
determine  whether  new  contract  language  or  a  revision  to  existing 
military  specifications  is  needed.  The  revision  or  expansion  of 
MIL-D-9412D  (Air  Force)  to  make  it  applicable  to  the  collection  of 
state-of-the-art  measurement  requirement  data  as  well  as  systems 
calibration  requirements  information  is  a  possible  solution. 

Experience  has  shown  that,  regardless  of  contract  method,  the 
forecast  of  new  measurement  requirements  is  difficult  to  achieve  and 
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often  delayed.  While  the  identification  of  metrology  problems  to  NBS 
and  the  service  metrology  centers  has  been  accomplished  by  rather 
informal  methods*  the  concern  here  is  with  early,  and  perhaps  more 
forma,.  identification,  so  that  the  lack  of  ability  to  measure  does  not 
become  a  deterrent  to  technological  progress  or  the  excuse  for  lack  of 
ability  to  determine  quality. 

Recommendation 

It  is  recommended  that  DoD  prepare  contract  language  and/or  a 
military  specification  to  require  systems  development  contractors  to 
identify,  analyze,  and  advise  the  metrology  engineering  centers, 
through  the  contracting  officers,  of  potential  new  state-of-the-art 
measurement  problems  early  in  the  development  phase  of  the  materiel. 


SELECTION  OF  PROPER  TEST  AND 
INSPECTION  EQUIPMENT  AND  MEASUREMENT  PROCESSES 


Inadequate  emphasis  is  being  placed  on  the  proper  selection  and 
support  of  teat  and  measuring  equipment  in  research,  development, 
design,  production,  quality  control,  and  operation  and  maintenance 
stages  and  in  monitoring  the  adequacy  of  test  equipment  and  its  usage 
in  research  and  development  activities. 

The  literature  and  technology  of  instrumentation  is  widely  scattered 
and  difficult  to  research;  hence,  the  selection  of  instruments  and  the 
techniques  for  their  application  are  equally  difficult.  Measurement 
needs  are  frequently  overwhelmed  by  "gold  plating"  or  are  only  par¬ 
tially  satisfied.  There  is  a  need  for  the  generation  and  publication  of 
technical  guidance  to  provide  assistance  for  optimizing  equipment 
selection. 

Developers,  development  agencies,  and  commodity  managers,  in 
their  zeal  to  deploy  military  equipment  to  meet  requirement  schedules 
all  too  frequently  fail  to  avail  themselves  of  the  metrology  competence 
of  standards  and  calibration  laboratory  personnel  and  similar  special¬ 
ists  in  the  selection  of  test  and  measuring  instruments  for  the  field. 

This  has  frequently  resulted  in  the  fielding  of  expensive,  overly  precise, 
or  overly  accurate  test  and  measuring  instruments. 

The  losses  due  to  unreliable  measuring  instruments  are  inestimable. 
DoD  decision  makers  must  have  reliable,  accurate  data  on  which  to  base 
their  conclusions  and  recommendations.  The  final  data  obtained  by 
testing  assembled  hardware  must  be  sufficiently  consistent  and  repeat- 
able  to  prove  the  confidence  level  of  reliability  estimates.  This  can 
only  be  achieved  through  provision  of  effective  equipment,  measure¬ 
ment  techniques,  and  data  feedback  systems. 


Recommendations 

It  is  recommended: 

1.  That  DoD  generrte  and  publish  a  technical  guide  for  use  by 
Government  and  contractor  engineering  personnel  to  aid  in  selecting 
test  and  inspection  equipment.  The  guide  should  emphasize  and/or 
require  the  use  of  metrology  and  calibration  personnel  to  aid  Govern¬ 
ment  and  contractor  research  and  design  engineering  personnel  in  test 
equipment  selection. 
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2.  That  a  requirement  be  placed  in  individual  service  regula¬ 
tions  or  existing  regulations  to  ensure  that  the  test  and  measureing 
instruments  of  research  and  development  activities  will  be  subjected 
to  the  same  quality  control  measures  as  are  those  in  other  phases  of 
the  materiel  life  cy.:le. 

3.  That  DoD  revise  appropriate  military  specifications  to  require 
controls  over  the  selection,  performance,  and  application  of  test  equip¬ 
ment  in  DoD  contracts. 

4.  That  DoD  develop  a  training  course  for  engineering  and  contract 
administration  personnel  in  the  selection  and  application  of  test  and 
measuring  equipment. 
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PROMULGATION  OF  UNIFORM  SPECIFICATIONS 
CONTROLLING  CONTRACTORS'  CALIBRATION  SYSTEMS 


Specifications  often  referenced  or  required  by  contracts  that 
delineate  calibration  requirements  are:  MIL-Q-9858A,  MIL-I-45208A, 
MIL-C-45662A,  MIL-C-55163  (Sig.  C),  MIL-Q-21549B  (WEP),  NASA 
NPC-200-1A,  NASA  NPC-200-2,  MIL-I-45607,  MIL-I-8500B  and  MIL- 
Handbook  50,  51  and  52.  These  specifications  differ  as  to  the  extent 
of  the  requirements  for  calibration  and  measurements. 

As  examples  of  confusion  created,  some  specifications  require 
adherence  to  a  stii.ct  10  to  1  accuracy  ratio,  while  others  make  no 
reference  to  such  ratios.  Some  specifications  are  not  clear  as  to  the 
requirements  for  recording  results  of  calibration.  Other  specifications 
allude  to  the  requirements  for  a  mandatory  recall  system  whereas  still 
others  are  specific. 

Due  to  the  problems  created,  Government  contract  evaluation 
agencies  find  it  difficult  to  enforce  and  administer  contractual  provisions 
as  envisioned  by  the  procuring  agencies. 

It  is  important  that  NASA  and  DoD  calibration  system  requirements 
be  consolidated  into  one  specification.  Many  R&D  contracts,  particu¬ 
larly  those  which  are  for  research  studies  or  do  not  require  fabrication 
of  hardware,  do  not  require  control  of  measuring  and  test  equipment. 

In  many  of  these  contracts,  measurement  data  represent  the  product, 
and  control  of  these  data  is  essential. 

There  are  measurements  being  made  of  physical  or  material 
phenomena  by  R&D  contractors  which  must  be  supported  by  measuring 
equipment  of  known  accuracies.  In  many  of  these  contracts,  measure¬ 
ment  data  represent  the  product  and  control  of  this  product's  quality 
ie  essential.  In  addition,  some  small  business  contracts, do  not  contain 
requirements  for  the  control  of  measurement  devices.  Small  business 
contractors  are  performing  measurements  of  products  with  devices 
which  must  also  be  of  known  accuracy. 

Recommendations 


It  is  recommended: 

1.  That  MIL-C-45662A  be  revised  and  adopted  as  the  stands rd 
calibration  specification  to  be  referenced  in  all  Government  contracts 
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including  small  business  and  R&O  contractors  where  measurements 
are  to  be  performed. 

2.  That  purchasing  officers  be  required  by  ASPR  or  directive  to 
include  applicable  portions  of  the  revised  specification  in  addition  to 
the  special  provisions  of  Standard  Form  32. 


PROVISION  OF  METROLOGY  SUPPORT  TO  CONTRACT 
ADMINISTRATION  PERSONNEL 


There  is  a  lack  of  understanding  of  the  need  for  calibration  and 
metrology  among  Government  and  contractor  personnel.  To  overcome 
thi»  lack  of  understanding,  it  is  recommended  that  an  orientation  film 
and  lecture  material  be  developed  for  presentation.  The  Navy  and  the 
Air  Force  both  have  films  which  were  developed  for  the  particular 
requirements  of  their  personnel.  These  films  have  proved  very  success 
ful  in  motivating  and  indoctrinating  personnel  in  the  specific  require¬ 
ments  for  calibration  and  metrology  ,vn  these  military  departments. 
These  films  could  be  used  as  guides  in  the  development  of  a  new  film 
for  presentation  on  a  wider  scale.  (AF  Film  SFP-1047  M USA F  Calibra¬ 
tion"  and  USN  Film  MN-10105  "Why  Calibrate"). 

The  calibration  and  measurement  specialists  iu  the  cognizant  con¬ 
tract  administration  organisations  are  trained  in  the  overall  evalua¬ 
tion  of  contractors'  programs.  However,  there  is  a  need  for  special* 
ized  support  and  training  in  certain  measurement  areas  ruch  as  optics, 
microwaves,  pressure,  flow  and  others.  Training  courses  are  avail¬ 
able  in  the  military  services  and  from  commercial  sources  in  most  of 
the  specialized  areas.  A  catalog  of  available  courses  should  be  com¬ 
piled  to  provide  a  ready  reference  for  determining  where  specific 
training  can  be  provided.  The  military  metrology  engineering  centers 
can  also  provide  specialized  training  and  technical  assistance  to  the 
cognizant  contract  administration  personnel. 

R  e  comm  ends  fons 

It  is  recommended: 

1.  That  DoD  develop  an  orientation  program  for  Government  and 
contractor  personnel  to  provide  general  knowledge  of  the  requirements 
for  a  calibration  and  metrology  program  in  the  contractors*  plants. 

2.  That  DoD  determine  specialized  training  requirements  for 
Government  calibration  and  measurements  personnel  to  improve  and 
augmeo;  their  capability  to  evaluate  contractors'  programs.  A  catalog 
of  available  calibration  training  courses  should  be  provided.  Whenever 
the  specific  capability  is  not  available  from  the  cognizant  contract 
administration  organization,  the  technical  assistance  of  the  military 
metrology  engineering  centers  should  bo  requested. 
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ESTABLISHMENT  OF  A  CENTRAL  POINT  OF  CONTACT 
WITHIN  OFFICE  SECRETARY  OF  DEFENSE  RESPONSIBLE 
FOR  OVERALL  POLICY  FOR  METROLOGY  AND 
CALIBRATION  AND  FOR  COORDINATION  WITH  OTHER 
GOVERNMENT  AGENCIES 


For  the  past  eight  years,  the  three  departments  and  AEC  have 
participated  in  combined  meetings  to  discuss  mutual  problems  concern¬ 
ing  metrology  engineering  and  calibration  services.  These  meetings 
contributed  to  the  important  resolution  of  several  problems  which 
interface  with  NASA,  AEC,  Department  of  Commerce,  and  other 
Government  agencies  requiring  coordination  at  the  OSD  level.  However, 
there  is  no  central  point  of  contast  within  OSD  responsible  for  overall 
policy  for  metrology  and  calibration,  or  through  which  DoD  components 
can  present  problems  involving  coordination  with  other  Government 
agencies.  There  needs  to  be  a  designated  jffice  within  OSD  to  provide 
general  policy  on  metrology  and  calibration  and  through  which  coordi¬ 
nation  can  be  attained  with  Government  agencies  outside  of  the  DoD. 

This  action  will  serve  *he  interest  of  OSD  as  well  as  the  three  depart¬ 
ments  and  DSA. 


Recommendation 

It  is  recommended  that  OSD  designate  a  specific  office  within  OSD 
(DDR&E,  ASD  (I&L),  possibly  the  DoD  Quality  and  Reliability  Council) 
for  policy  direction  and  to  serve  as  a  central  point  of  contact  for  metrol¬ 
ogy  and  calibration  matters. 
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REPORT  OF  PANEL  9 


TITLE:  Quality  of  Technical  Data 
OBJECTIVE: 

To  improve  the  DoD  capability  to  provide  data  to  users  that 
are  suitable  for  the  intended  purpose,  at  lowest  practical  cost. 

TOPICS  DISCUSSED: 

1.  Role  of  the  Contractor  in  Data  Quality 

2.  Government  Responsibilities  in  Data  Quality 

3.  Training 

4.  Legibility 

5.  Data  Warranty 

6.  Engineering  Data  "Defects"  and  "Rights" 

7.  Lack  of  Uniformity  in  Data  Requirements 

8.  Third  Party  Evaluation  of  Data 

9.  Application  of  Zero  Defects  to  Data 

10.  Industry  Viewpoints  on  Data  Adequacy 

11.  Use  of  Data  in  Relationship  to  Quality  Techniques 

12.  Matching  Data  against  Hardware 
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Lt.  Col.  John  W.  Barlow,  USAF,  HQS,  AFSC 

Mr.  D.  Capretti,  HQS,  Naval  Material  Command 

Mr.  B.  D.  Weintraub,  HQS,  DCAS 

Col.  Samuel  T.  Jacks, 

Defense  Electronics  Supply  Center 

Mr.  Robert  L.  Body,  Kennedy  Space  Center 

Mr.  I.  Appelblatt,  Army  Mobility  Command 

Mr.  Norman  Miller,  Frankford  Arsenal 

Mr.  F.  J.  Schiavi, 

Naval  Ordnance  Systems  Command 

Mr.  Cornelius  J.  Brown 
Federal  Supply  Service 


Lt.  Col.  William  C.  Shepherd 

Office,  Asst.  Secretary  of  Defense  (I&L) 


Mr.  Donald  R.  Mitchell 

Office,  Asst.  Secretary  of  Defense  (I&L) 


Mr.  Irving  Altman, 

Office,  Asst.  Secretary  of  Defense  (I&L) 


ROLE  OF  THE  CONTRACTOR  IN  DATA  QUALITY 


Basic  quality  assurance  policy  of  DoD  Instructions  4155.  6  and 
4155.  8  establish  the  contractors  responsibility  for  assuring  that  supplies 
conform  to  contractual  requirements  and  for  performing  all  inspections 
and  tests  called  out  in  the  contract.  These  policies  are  well  suited  to 
data.  A  key  to  better  data  quality  lies  in  further  amplifying  and 
enforcing  contractor  quality  assurance  practices.  Adequate  drawing 
quality  and  content  and  assuring  that  data  accurately  reflects  hardware 
produced  is  a  responsibility  of  the  contractor. 

Although  MIL -Q -9858  requires  that  contractors  establish  a  quality 
program,  it  contains  very  little  criteria  on  what  the  minimum  consti¬ 
tuent  elements  or  a  data  control  program  must  be.  Consequently,  the 
government  has  little  basis  for  criticizing  or  disapproving  a  contractors 
data  quality  program,  and  offers  minimum  guidance  to  a  government 
inspector  or  other  government  personnel  who  review  such  a  program. 


Recommendation 

It  is  recommended  that  DoD  develop  and  specify  the  minimum 
elements  of  an  acceptable  contractors  data  quality  program  in  all 
contracts  requiring  data  delivery.  These  requirements  should  be 
placed  in  a  military  specification,  preferably  in  MIL -Q -9858,  (Appendix 
A  provides  recommendations  for  the  minimum  elements  to  be  included 
in  a  contractors  data  quality  program). 
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GOVERNMENT  RESPONSIBILITIES  IN  DATA  QUALITY 


The  DoD  has  several  basic  responsibilities  that  affect  directly  the 
utility  of  the  data  product  for  its  purpose: 

1.  Formulating  data  requirements  that  meet  but  do  not  exceed 
the  requirements  of  users. 

2.  Communicating  these  requirements  to  the  contractor  clearly 
and  concisely,  and  ensuring  that  he  understands  them. 

3.  Developing  system  requirements  for  data  quality  control. 

4.  Developing  procedures  for  audit/review  of  contractors  QA 
Program  for  technic.'.!  data  control. 

5.  Accepting  of  final  data  product  based  on  contract  requirements. 

6.  Assuring  the  proper  use  of  data  acquired. 

The  force  of  DoD's  technical  data  program  is  directed  toward  the 
formulation  of  data  requirements  and  communicating  them  to  the  con¬ 
tractor.  Therefore,  this  panel  did  not  consider  these  aspects  of  the 
problem  in  detail.  This  panel  feels  that  the  area  in  need  of  most 
improvement  lies  in  assuring  that  the  contractors  clearly  understand 
the  data  requirement,  and  that  the  execution  of  his  data  quality  program 
is  effectively  monitored  by  specialists  in  the  fields  that  are  affected  by 
the  data  product.  These  specialists  might  include  engineering,  procure¬ 
ment,  maintenance,  drafting,  production,  materials,  standardization, 
quality  assurance,  and  other  types  of  personnel. 

Present  directives  dealing  with  provisioning  must  insure  recording 
of  procurement  method,  to  justify  the  method  recorded,  and  provide 
timely  intra -government  exchange  of  current  technical  data  to  the 
purchasing  agency  to  insure  timely  customer  support. 


Recommendations 


It  is  recommended: 

1.  That  DoD  Instruction  5010.  12  be  modified  to  include  a  detailed 
statement  of  Government  responsibilities  relative  to  acquisition  of 
technical  data,  i.  e.  review,  evaluation  of  contractors  quality  program 
for  control  of  Technical  Data. 
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2.  That  DoD  develop  a  handbook  similar  to  MIL  Handbooks  50, 

51  and  52,  outlining  procedures  for  review  of  the  adequacy  of  con¬ 
tractors  technical  data  control  systems. 

3.  That  DoD  revise  appropriate  provisioning  regulations  to  require 
that  supply  support  requests  record  the  method  of  purchase,  provide 
necessary  justification,  and  include  technical  data  suitable  for  the 
method  of  purchase  specified.  Page  3-14,  Attachment  II,  Enel.  3, 

of  Joint  Regulation  issued  under  No's  DSAR  4140.  35,  AR  710-25,  AFR 
67-8,  BUSANDAINST  4423.  10,  MCC  4423.9  requires  revision  to 
implement  this  recommendation. 


TRAINING 


Competency  in  determining  quality  aspects  of  technical  data  to 
contractual  requirements  is  dependent  upon  qualified  and  experienced 
government  personnel.  Experience  has  demonstrated  that  many 
personnel  concerned  with  quality  assurance  acceptance  functions  are 
in  need  of  specialized  training  in  data  management  techniques  and 
specifications  requirements  to  assure  that  contractor's  program 
results  in  preparation  of  data  to  contractual  requirements. 


{  Recommendation 

III  is  recommended  that  a  training  syllabus  be  developed  that  will 
cover  the  essentials  required  for  government  personnel  concerned 
]  with  data  quality  assurance.  An  appropriate  training  syllabus  might 

\  include  the  following  subjects: 

1.  Coverage  of  material  essential  to  the  functions  of  the  QA 
technical  data  specialist. 

2.  Factual  information  for  formulating  decisions  and  interpre¬ 
tations  of  specification/ standard  requirements. 

3.  Coverage  of  technical  data  vocabulary  and  meanings. 

4.  Importance  of  procuring  technically  accurate  and  adequate 

data. 


5.  Inspection/ acceptance  of  data. 

6.  Inspection  techniques. 
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LEGIBILITY 


Technical  data  acquired  by  the  DoD  are  often  illegible  and  are  not 
capable  of  producing  legible  microfilm  or  copies  for  support  of  intended 
functions. 

The  principal  specifications  and  related  documents  developed  to 
ensure  legible  technical  data,  MIL -D -1000  for  Engineering  Drawings 
and  Associated  Lists,  MIL-M-9868  for  microfilm,  MIL-D-5480  for 
drawing  reproduction,  and  MIL -HD BK -303  which  gives  guidance  for 
microfilming  are  sound.  The  proper  use  and  consciousness  of  these 
specifications,  to  a  great  degree,  have  reduced  the  problem  of  illegi¬ 
bility,  particularly  during  the  phase  of  preparation  of  original  technical 
data. 

However,  data,  not  prepared  or  reproduced  under  such  finite 
controls  continue  to  be  an  important  problem.  Many  engineering 
drawings  prepared  before  microfilming  became  generally  practiced 
are  not  suitable  for  microfilming  or  reproduction.  A  major  part  of 
the  problem  rests  with  the  preparation  of  microfilm  aperture  cards 
and  other  type  reproduction  from  illegible  originals  and  first  generation 
microfilm  therefrom  and  the  subsequent  use  of  these  aperture  cards  in 
procurement  bid  sets. 


Recommendations 


It  is  recommended: 

1.  That  existing  technical  data  (engineering  drawings)  which  are 
illegible  and  necessary  to  support  DoD  assigned  functions  be  considered 
for  restoration  through  the  use  of  phototracing  and  photorestoration 
processes.  The  preparation  of  new  drawings  or  redraws  should  be 
accomplished  as  a  last  resort. 

2.  That  in  the  preparation  of  new  technical  data,  the  Services 
continue  emphasis  on  inspection  by  contractor  and  verification  by 
government  skilled,  and  technically  trained  personnel  for  legibility  and 
reproducibility  in  accordance  with  specified  and  mutually  understood 
governing  specifications.  These  actions  should  take  place  during  all 
phases  of  preparation  and  subsequent  processing  and  handling  of 
technical  data. 
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DATA  WARRANTY 


The  "data  withholding  of  payment"  clauses  of  ASPR  9-207.  2  provide 
for  withholding  of  monies  for  failure  on  the  part  of  the  contractor  to 
make  timely  deliveries  of  data  or  because  the  data  offered  for  acceptance 
is  deficient.  If  deficient  data  are  accepted  by  the  Government,  such 
acceptance  legally  relieves  the  contractor  of  his  contractual  obligation 
to  complete  or  correct  the  data.  Inaccurate  and  incomplete  data  may 
prevent  the  Government  from  utilizing  the  data  for  the  purpose  for 
which  it  was  procured,  including  competitive  reprocurement. 

The  scope,  complexity  and  sheer  volume  of  scientific  and  technical 
data  causes  a  thorough  review  by  qualified  Government  personnel  of 
delivered  data,  before  acceptance,  to  be  administratively  impractical. 

On  the  other  hand,  the  contractor  is  not  only  contractually  bound,  but 
also  is  uniquely  qualified  by  reason  of  intimate  knowledge  of  his  own 
methods  of  manufacture,  processes,  and  know-how  to  undertake  the 
burden  of  assuring  compliance  with  his  contract  data  requirements. 

Based  upon  this  premise,  an  Air  Force  -  Industry  team  considering 
this  question  has  prepared  a  warranty  clause  for  inclusion  in  DoD 
contracts.  This  clause  guarantees  to  the  Government  that  the  data 
the  contractor  delivers  is  complete  and  accurate  in  accordance  with  his 
contractual  data  requirements.  The  clause  also  requires  the  contractor 
to  supply,  correct,  or  replace  the  deficient  data  provided  he  is  notified 
of  the  deficiency  within  a  specified  period  of  time  after  the  data  is 
delivered. 


Recommendation 

It  is  recommended  that  the  ASPR  Committee  favorably  consider 
the  data  warranty  clause  jointly  developed  by  Air  Force  and  Industry. 
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ENGINEERING  DATA  "DEFECTS"  AND  "RIGHTS" 


The  major  deficiencies  in  the  compliance,  by  contractors  and 
vendors,  with  drafting  standards,  accounting  for  approximately  75%  of 
the  data  rejected  by  one  military  service  are  as  follows:  (1)  incorrect 
drawing/part  number  representation;  (2)  material  processes  not  identi¬ 
fied;  (3)  contractors  and  vendors  documents  referenced  on  drawings 
not  submitted;  (4)  dimensions/tolerances  omitted;  (5)  improper  use  of 
proprietary  legends.  Any  combination  of  the  above  deficiencies  has 
been  called  "swiss-cheese"  drawings.  There  is  evidence  that  a  sub¬ 
stantial  portion  of  these  deficiencies  are  attributable  to  the  effort  of 
contractors  to  prevent  "so  called"  proprietary  information  from  being 
disclosed  to  potential  competitors. 

All  five  deficiencies  have  a  major  impact  on  logistics  operations. 

The  primary  impact  of  "rights"  is  relative  to  the  procurement  process 
and  applicable  only  to  a  relatively  small  number  of  items  or  drawings. 

To  a  large  extent  this  process  is  complicated  by  the  necessity  on  the 
part  of  the  "prime"  contractors  to  flow -down  data  rights  provisions  to 
vendors. 

The  base  of  "intended  usage"  provided  in  Mil-D-1000  coupled  with 
ASPR  Data  Rights  revision,  provide  an  improved  frame  of  reference 
with  respect  to  data  quality.  ASPR  revisions  now  permit  the  government 
to  recognize  a  "limited  use"  for  privately  developed  data.  When  the 
data  requirements  are  for  a  reprocurement  mission  ASPR  revision  per¬ 
mits  specific  negotiation  for  the  purchase  of  privately  developed  data 
for  competitive  procurement  purposes. 


Recommendation 

It  is  recommended  that  current  contracts  he  reviewed  by  the  Services, 
and  when  economically  practical,  the  current  ASPR  provisions  be 
applied  to  appropriate  data  requirements. 
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LACK  OF  UNIFORMITY  IN  DATA  REQUIREMENTS 


The  need  for  completeness  and  elaboration  of  technical  data 
increases  directly  with  expansion  of  the  distance  between  the  designer 
and  the  production  shop.  The  proportionality  also  exists  with  respect 
to  the  number  of  contractual  and  administrative  barriers  between  the 
man  on  ;he  drafting  board  and  the  man  on  the  man  on  the  drafting  board 
and  the  man  on  the  machine. 

A  small  shop  in  which  :he  designer  is  personally  able  to  convey 
instructions  to  the  machinist,  etc. ,  by  word  of  mouth  needs  much  less 
in  the  way  of  technical  data  than  does  a  large  integrated  industry  with 
many  suppliers  and  automated  machinery.  These  n  leds,  which  directly 
reflect  their  ability  to  control  the  design  and  quality  of  product,  deter¬ 
mine  elaboration  of  data.  This  has  resulted  in  the  proliferation  of 
parochial  drafting  standards  and  untold  variations  from  standards. 

Current  government  contracting  practices  are  such  that  govern¬ 
ment  agencies'  needs  for  completely  defined  drawing  information  in 
many  instances  exceeds  that  of  industry.  In  addition,  in  spite  of 
extensive  standardization  efforts,  the  different  military  services  impose 
differing  requirements  for  essentially  the  same  needs,  which  vary  in 
accordance  with  the  parochial  differences  of  their  major  supplier 
industries. 

Latitude  must  be  provided  for  each  service  to  meet  needs  peculiar 
to  it.  However,  it  is  felt  that  an  optimum  degree  of  standardization 
has  not  been  attained.  Greater  uniformity  of  practices  would  greatly 
reduce  data  quality  problems,  especially  as  they  pertain  to  vendors. 

The  advantages  of  the  contractor  and  the  government  communicating  in 
identical,  completely  understandable  technical  language,  which  is  after 
all,  all  that  a  drawing  package  is  supposed  to  do,  are  many.  If  we  had 
such  identical  practices  initially  we  would  not  have  to  pay  a  premium 
for  drawings  initially  made  or  redrawn  to  our  requirements. 

We  would  also  save  considerable  time,  avoid  expensive  misunder¬ 
standings,  and  better  utilize  scarce  engineering  resources. 

The  benefits  of  such  a  program  are  significant  enough  to  be  trans¬ 
lated  to  a  national  scale  for  the  eventual  improvement  of  our  nation's 
technology. 

Panel  9  recognizes  that  the  DoD  has  for  a  number  of  years  directed 
programs  to  bring  about  greater  uniformity  in  drafting  practices.  The 
panel  endorses  these  efforts  but  believe  they  should  be  accelerated. 
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Recommendations 

It  is  recommended: 

1.  That  the  DoD,  under  the  monitorship  of  the  Office  of  Technical 
Data  and  Standardization  continue,  and  accelerate  to  the  extent  possible, 
programs  to  bring  about  greater  uniformity  in  the  drawing  practices 

of  the  military  services. 

2.  That  DoD  accelerate  its  efforts  and  provide  leaderchip  to 
develop  and  promote  national  standardization  of  drawing  practices  for 
the  purpose  of  eventually  adopting  a  national  system  of  drawing  prac¬ 
tices.  Consideration  should  be  given  to  use  to  the  Federal  Series  of 
documents. 


THIRD  PARTY  EVALUATION  OF  DATA 


It  has  been  suggested  often  that  the  DoD  might  overcome  the 
pioblem  of  data  quality  by  employing  a  "third  party"  contractor  to 
evaluate  the  data  produced  by  prime  contractors  before  acceptance  by 
the  government. 

Based  on  review  of  experience  and  informed  opinion  with  regard 
to  use  of  "third  party"  contractors,  Panel  9  feels  that  the  value  of  such 
use  is  limited.  To  obtain  proper  results,  a  data  review  activity  must 
have  direct  experience  and  current  knowledge  associated  with  the 
development  itself. 

There  is  always  some  gain  in  accuracy  and  completeness  resulting 
from  any  comprehensive  review.  To  the  extent  that  DoD  activities 
might  be  limited  in  their  capacity  to  perform  a  comprehensive  review, 
"third  party"  contractors  can  be  employed  advantageously. 


F  ecommendation 

It  is  recommended  that  "third  party"  contractors  be  employed 
only  after  careful  consideration  on  a  case  by  case  basis  where  limited 
in-house  capability  prevents  adequate  review  of  data.  Such  contractors 
shoo'd  net  be  employed  for  evaluation  of  technical  content  of  data 
except  under  unusual  circumstances. 


230 


APPLICATION  OF  ZERO -DEFECTS  TO  DATA 


The  concept  of  Zero  Defects  (ZD)  has  not  been  fully  exploited  in 
the  technical  data  area.  In  cases  where  ZD  has  been  applied  to  data, 
beneficial  results  are  at  least  as  great  as  those  achieved  with  hardware. 
An  article  in  the  Wall  Street  Journal  on  6  April  1965  quoted  dramatic 
reduction  of  drawing  errors  by  the  General  Electric  Company  after 
application  of  the  ZD  concepts. 

Panel  9  feels  that  no  attempt  should  be  made  to  initiate  a  separate 
ZD  program  for  data,  but  that  more  can  be  done  to  insure  that  ZD 
concepts  are  applied  to  data  wherever  ZD  is  used. 


Recommendation 

It  is  recommended  that  the  Services  introduce  technical  data  con¬ 
siderations  in  all  ZD  activities. 
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INDUSTRY  VIEWPOINTS  ON  DATA  ADEQUACY 


The  Office  of  Technical  Data  and  Standardization  Policy  is  fully 
committed  to  the  practice  of  working  closely  with  Industry  on  all  policy 
matters  within  its  purview.  DoD's  files  are  replete  with  Industry 
association  reports  and  studies  on  many  aspects  of  data  acquisition, 
preparation,  handling  and  retrieval.  Many  of  these  reports  have  led 
to  major  improvements  in  DoD  policies  and  practices.  However,  some 
industry  recommendations  are  so  general  as  to  be  of  little  use,  some 
deal  with  detailed  drafting  techniques,  but  none  deals  specifically  and 
broadly  with  data  quality. 

Recently  a  selected  group  of  knowledgeable  and  respected  individuals 
in  a  variety  of  industries  was  queried  on  this  particular  subject.  The 
replies,  made  frankly  and  in  the  spirit  of  helpfulness,  represent  a 
cross-section  of  American  Industry.  There  is  some  disagreement 
among  them  and  certainly  many  of  their  remarks  would  be  challenged 
by  various  elements  of  DoD.  Nevertheless,  they  are  the  well-thought- 
out  viewpoints  and  observations  of  responsible  managers  and  operating 
officials  who  live  daily  with  the  problems  of  creating  technical  data 
and  complying  with  our  voluminous  and  often  inconsistent  requirements. 

In  the  interest  of  providing  a  balanced  documentation  of  the  subject 
of  data  quality,  the  letter  which  requested  the  comments  and  a  sampling 
of  the  replies  have  been  included  in  Appendix  B.  Since  these  were 
personal  replies  to  a  personal  letter,  identification  of  individuals  and 
their  affiliation  have  been  deleted.  The  authors’  permission  to  reproduce 
their  letters  is  gratefully  acknowledged. 


Recommendation 

It  is  recommended  that  the  thoughts  expressed  in  these  and  the 
other  letters  received  be  objectively  considered  by  those  in  the  DoD 
components  who  are  responsible  for  data  policy  and  requirements. 
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USE  OF  DATA  IN  RELATIONSHIP  TO  QUALITY  TECHNIQUES 


Quality  of  engineering  data  involves  considerations  from  configura¬ 
tion  management  thru  the  cycle  of  mission  usage  to  disposal.  Sampling 
of  the  compliance  with  drafting  standards  indicates  that  approximately 
7%  is  defective.  These  defectives  have  a  serious  effect  on  logistics 
operations.  Of  the  millions  of  drawings  received  each  year  it  is  esti¬ 
mated  that  the  probability  of  use  for  reprocurement  lies  within  5%. 
Generally,  this  use  is  the  most  demanding  one  for  comprehensive 
detail. 

The  DoD  policy  of  ''mission  orientation"  is  outlined  in  MIL-D-1000 
and  further  defined  in  the  "ordering"  techniques  of  the  services. 

Related  programs  such  as  MIL-STD-789  (ASG),  "Procurement  Method 
Coding  of  Aeronautical  Replenishment  Spare  Parts"  provides  for  item 
by  item  determination  and  the  acquisition  of  data  appropriate  to  the 
method  of  procurement  intended. 

This  identification  or  selection  of  spares  and  repair  parts  for 
competitive  procurement  during  the  acquisition  cycle  provides  a  listing 
of  items  that  could  be  used  to  provide  for  the  application  of  special 
Quality  Assurance  techniques  for  such  items. 


Recommendation 

It  is  recommended  that  the  current  project  to  expand  such  techniques 
as  provided  by  MIL-STD-789  (ASG)  to  a  DoD  w?de  document  be  expe¬ 
dited.  Quality  assurance  techniques  should  provide  special  considera¬ 
tion  for  items  designated  for  competitive  procurement. 
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MATCHING  DATA  AGAINST  HARDWARE 


Data  is  a  basic  management  tool  which  provides  the  vital  l  .nk 
essential  to  maintenance  of  a  closed  loop  effort  between  the  conceptual 
definition,  acquisition  and  operational  phase  of  any  significant  weapons 
system  program.  There  is  a  logical  progression  by  which  the  pervasive 
influence  of  data  extends  to  embrace  configuration  management.  When 
data  for  competitive  procurement  is  applied  and  used  in  an  orderly 
fashion,  following  a  defined  series  of  procedural  steps  and  decision 
rules  to  end  in  a  design  decision,  a  configuration  has  been  established. 
The  First  Article  Configuration  Review  (FACI)  or  configuration  audit 
review  (CAR)  is  a  technical  audit  comparing  prototype  hardware  with 
the  technical  data,  drawings  and  specifications,  to  be  used  in  describing 
the  production  baseline.  When  prototype  hardware  has  been  produced 
the  technical  data  of  this  unit  shall  be  compared  directly  with  the 
as -built  configuration  of  the  same  unit.  The  as -built  unit  will  be  com¬ 
pared  directly  with  the  engineering  drawings  to  the  level  of  design  dis¬ 
closure  specified  on  the  drawings  and  related  data  assembled  by  the  top 
drawing  specified  in  the  description.  The  DD  Form  250  (Material 
Inspection  and  Receiving  Report)  will  not  be  completed  until  differences 
in  documentation  and  as -built  configuration  are  corrected. 

The  best  way  to  improve  the  quality  of  a  technical  data  package  for 
competitive  procurement  package  is  to  use  it  in  the  manufacture  of  a 
limited  production  quantity  before  undertaking  full-scale  production. 

This  limited  production  experience  will  shake  out  the  numerous  errors 
and  omissions  introduced  during  development  of  the  technical  data 
package.  Subsequent  field  testing  or  service  evaluation  of  this  limited 
quancity  will  reveal  additional  weaknesses  not  detected  during  earlier 
prototype  tests,  or  those  that  are  propagated  by  unfamiliar  personnel 
usage  or  fleet/field  abuse.  The  feedback  from  fleet/field  use  when 
included  with  the  limited  production  experience  will  provide  a  package 
suitable  for  full-scale  production.  Although  the  limited  production  will 
net  eliminate  all  of  the  changes  that  will  be  found  necessary  during  the 
first  full-scale  production  run,  it  will  substantially  improve  the  condi¬ 
tion.  With  subsequent  procurements,  tho  number  of  necessary  changes 
will  continue  to  decline  as  the  quality  of  the  technical  data  package 
continue'  to  improve  with  maturity.  Unfortunately,  the  best  technical 
data  package  for  production  is  the  one  no  longer  needed  at  the  end  of 
the  production  period. 

The  validity  of  technical  data  varies  from  that  which  has  been 
proven  by  having  single  units  produced  to  that  resulting  from  corrections 
identified  by  limited  production  quantity  field/fleet  use. 


Recomm  endations 


It  is  recommended: 

1.  That  validation  of  drawings  be  accomplished  by  building  testing 
and  obtaining  feedback  from  the  maximum  number  of  items  practical. 

2.  That  the  Services  include  requirements  in  the  contractors' 
data  control  programs  for  configuration  audit  review  to  insure  that 
data  delivered  matches  the  hardware. 
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ELEMENTS  OF  A  CONTRACTORS  DATA  QUALITY  PROGRAM 


The  DoD  document  recommended  should  establish  at  least  the 
following  requirements  for  a  quality  program  to  provide  technical  data 
to  the  requirements  of  the  contract: 

1.  Contractor  shall  maintain  an  effective  and  economical  quality 
program  for  technical  data,  which  is  planned  in  consonance  with  other 
administrative  and  technical  programs.  The  program  including  pro¬ 
cedures,  shall  encompass  all  aspects  of  technical  data  quality  during 
all  phases  of  preparation  (plar  ling,  outlining,  drafting,  change  control, 
writing,  reviewing,  validation,  etc. ).  The  extent  of  the  program 
established  shall  be  based  on  the  complexity  of  the  hardware  covered 
and  the  interchangeability  and  reliability  requirements  thereof.  The 
program  shall  assure  that  adequate  use  of  technical  data  is  maintained 
throughout  all  areas  of  affected  contract  performance  and  shall  provide 
for  the  prevention  and  ready  detection  of  discrepancies  and  for  timely 
positive  corrective  action.  The  contractor  shall  maintain  and  make 
available  to  the  government  representative  objective  evidence  of 
technical  data  conformance  to  contractual  requirements. 

2.  Procedures  required  shall  include,  but  not  be  limited  to,  those 
providing  for  the  inspection  and  approval  of  technical  data  for  scope, 
style,  format,  legibility,  reproducibility,  technical  accuracy  and 
adequacy,  and  the  presence  of  required  instructions  and  information. 
The  procedures  shall  provide  the  method,  place  and  organization 
responsible  for  the  performance  of  technical  data  inspections,  reviews 
and  validation  exercises. 

3.  The  contractor  shall  establish  and  maintain  control  at  located 
points  in  the  preparation  and  production  of  technical  data  to  assure 
continuous  control  (in-process  inspection)  of  accuracy  and  adequacy  of 
data  and  their  total  conformance  to  contractual  requirements.  The 
controls  instituted  shall  be  suited  for  the  type  of  procurement  defined 
by  the  contract  or  order. 

4.  Measuring  and  testing  devises  used  in  validating  and  verifying 
technical  data  shall  be  calibrated  against  measurement  standards  on 
designated  measuring  equipment  traceable  to  the  NB5,  at  established 
periods  to  assure  continued  accuracy.  The  contractor  shall  prepare 
and  maintain  a  written  schedule  for  the  maintenance  and  calibration  of 
such  equipment  based  on  type,  purpose  and  degree  of  usage. 
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5.  Drawing  and  change  control  features  of  paragraph  4  of 
MIL-Q-9858A  should  be  made  applicable  to  change  control  of  all 
pertinent  data  required  by  contracts. 

6.  The  quality  control  program  shall  delineate  procedures  and 
their  controls  that  will  produce  adequate  technical  data  for  delivery 
v  ithin  contractual  schedules. 

7.  The  quality  program,  including  procedures,  shall  be  docu¬ 
mented  and  subject  to  the  review  by  the  local  government  representative. 
The  procedures  shall,  including  listings  of  contractor  organization,  be 
responsible  for  check  and  control  points  through  the  phases  of  technical 
data  development,  validation,  verification  and  production. 

8.  Records  of  all  inspections,  reviews,  validations,  verifications 
and  corrective  actions  performed  by  the  contractor  shall  be  kept  comp¬ 
lete  and  available  to  the  government  during  the  performance  of  the 
contract. 
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Appendix  B 


OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 
Washington,  D.  C.  20301 

INSTALLATIONS  AND  LOGISTICS  9  May  1966 

Determining  the  adequacy  of  data  is  one  of  the  fundamental  concerns 
of  this  Office.  To  date,  we  have  moved  fairly  slowly  in  this  area  due  to 
the  pressure  of  other  problems.  We  now  feel,  however,  that  the  time 
has  come  to  identify  more  thoroughly  the  nature  of  this  problem  and  to 
consider  what  actions  need  to  be  taken. 

To  this  end,  I  have  established  a  military  group  to  study  and  to 
recommend  actions  to  improve  data  quality,  starting  with  drawings.  In 
addition,  a  project  has  been  initiated  to  strengthen  the  quality  assurance 
privisions  of  MIL -D -1000.  Both  of  these  efforts  are  just  getting  under 
way. 


Experience  to  date  indicates  that  there  are  many  instances  where 
the  quality  of  drawings  delivered  to  the  government  is  not  up  to  specifi¬ 
cations.  We  do  not  know  as  yet  how  general  this  condition  is,  but  we 
do  know  that  in  cases  where  sampling  inspection  has  been  used  as  the 
basis  for  acceptance,  the  quality  of  drawings  has  risen  sharply.  There¬ 
fore,  we  have  been  seriously  considering  requiring  the  use  of  sampling 
techniques  for  acceptance  of  drawings  delivered  under  MIL -D -1000. 

We  have  received  comments,  however,  which  indicate  that  sampling 
procedures  should  not  be  made  applicable  to  drawings. 

Generally,  we  believe  a  quality  of  drawings  analysis  can  be  broken 
into  two  main  divisions:  (1)  technical  and  (2)  format.  The  former 
pertains  to  the  degree  to  which  the  drewing  reflects  the  product  produced, 
and  the  latter  pertains  to  the  relationship  of  the  drawing  to  "design 
disclosure"  and  format  requirements  of  MIL-D-1000  and  MIL-STD-100. 
There  appears  to  be  a  tendency  to  place  format  adequacy  in  a  role  of 
secondary  importance  to  technical  adequacy.  With  respect  to  mundane 
requirements  such  as  the  size  of  title  block,  this  a  valid  observation. 

But,  to  the  extent  that  drawings  omit  needed  information  or  express 
information  in  terms  of  company  practices,  drawings  may  well  not  be 
usable  by  the  government  if  format  requirements  are  not  met. 

In  view  of  the  above,  I  would  appreciate  your  comments  on  the 
major  aspects  of  thi$  problem  and  specifically  with  respect  to: 
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1.  Whether  or  not,  to  your  knowledge,  the  DoD  has  a  general 
quality  problem  in  connection  with  drawings  it  acquires. 

2.  What  actions  would  you  recommend  to  improve  the  quality  of 
drawings? 

3.  To  what  extent,  if  any,  quality  assurance  or  inspection 
personnel  should  become  involved  in  checking  drawing  quality? 

4.  Should  sampling  techniques  be  used? 

5.  Can  we  rely  on  in-process  controls  on  drawing  quality  to  the 
exclusion  of  a  thorough  final  inspection? 

To  my  mind,  the  assurance  of  data  quality  ranks  equally  with  the 
cardinal  problems  of  data  requirements,  rights  in  data,  and  the 
storage/ retrieval  problems.  I  shall,  therefore,  be  looking  forward 
to  your  thoughts  on  this  subject. 


Sincerely, 


(Signed)  O.  C.  Griffith 
Colonel,  USAF 

Acting  Director,  Office  of  Technical 
Data  and  Standardization  Policy 
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June  20,  1966 


Colonel  O.  C.  Griffith 
Acting  Director,  Technical  Data  and 
Standardization  Policy,  Office  of  the 
Assistant  Secretary  of  Defense  (X&L) 

Cameron  Station,  Building  3 
Alexandria,  Virginia 

Subject:  Adequacy  of  Data 

Dear  Ole: 

In  the  past,  I  have  discussed  with  you  my  viewpoints  on  a  sampling 
procedure  to  review  the  quality  of  data  to  be  submitted  to  the  customer. 
I  have  forwarded  to  you  copies  of  our  procedure  and  charts  showing 
progress  in  improving  output  within  our  department.  We  have  found 
this  procedure  to  be  very  beneficial  for  many  reasons.  Naturally, 
from  the  experience  of  the  past  three  years,  I  am  in  favor  of  a  con¬ 
trolled  quality  assurance  audit  or  sampling  inspection  prior  to  sub¬ 
mission  to  the  DoD. 

Specifically  as  to  your  questions: 

1.  From  my  past  experience,  serving  on  committees  to  the 
various  services  in  DoD,  I  have  come  across  many  circumstances 
where  DoD  had  a  quality  problem.  From  these  circumstances  and  our 
own  experience,  I  believe  that  they  have  a  general  quality  problem.  I 
think  this  problem  places  a  conscientious  contractor  in  an  unfavorable 
position  on  competitive  bids.  To  assure  quality  documentation,  a 
system  of  control,  which  does  affect  costs,  is  required. 

2.  I  recommended  that  the  services  review,  on  a  sampling  basis, 
and  not  necessarily  in  accordance  with  MIL-STD-105,  the  output  of  a 
company  to  assure  that  within  reason  the  documentation  meets  the 
requirements.  Naturally,  from  our  own  experience,  I  believe  that  our 
quality  assurance  program  could  be  adapted  by  the  military  and  assure 
quality  documentation. 

3.  I  do  not  believe  that  quality  assurance  or  inspection  personnel, 
as  we  think  of  quality  control  personnel,  should  become  involved  in 
checking  drawing  quality.  Within  our  department,  we  check  practically 
100  percent  of  our  output  by  what  we  call  "career  checkers",  which  is 
our  highest  drafting  classification.  However,  as  mentioned  before,  we 
do  have  a  quality  audit  program,  in  which  we  have  two  to  four 
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prov  isional  auditors  review,  on  a  sampling  basis,  the  data  package 
from  the  1423  input  through  the  microfilm  output. 

4.  Yes,  in  accordance  with  the  enclosed  report. 

5.  No,  I  do  not  believe  so.  I  believe,  no  matter  how  conscientious 
and  competent  a  company  is,  there  has  to  be  some  quality  assurance 
audit  by  the  customer.  This  will  vary  depending  upon  the  past  per¬ 
formance,  but  should  not  be  excluded. 
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June  17,  1966 
Cl -75-50/FS 


Col.  O.  C.  Griffith 
Acting  Director 
Office  of  Technical  Data  and 
Standardization  Policy 
Department  of  Defense 
Building  3  -  Room  BilO 
Cameron  Station 
Alexandria,  Virginia  20301 

Dear  Ole: 

About  1954-55  the  first  attempt  was  made  to  include  data,  as  such, 
under  the  quality  control  Specification  MIL-Q-5923.  Sines  then  there 
has  been  great  amount  of  argument  and  difficulty  over  what  appeared 
to  us  to  be  a  relatively  simple  matter.  This  is  not  to  say  that  there 
were  no  problems  associated  with  data  quality.  Primarily,  such 
problems  concerned  the  legibility  of  the  reproduction  furnished,  although 
some  other  problems  concerning  the  completeness  of  data  and  the 
proprietary  rights  thereto  were  also  involved.  Approximately  95%  of 
these  other  problems  occurred  (and  still  do  occur)  in  the  area  of  vendor 
data.  We  estimate  vendor  data  to  comprise  at  least  50%  of  the  overall 
data  package,  and  a  far  greater  percentage  of  the  total  data  problem. 

Data  displays  design  and  manufacturing  information  and  instructions, 
and  the  adequacy  of  those  instructions  cannot  be  demonstrated  as  a 
function  of  quality  control  of  data.  The  adequacy  of  design  and  manu¬ 
facturing  instructions  is  subject  to  review  and  checking  within  the  con¬ 
tractor’s  engineering  effort  and  is  demonstrated  by  the  eventual 
production  of  hardware  and  its  appropriate  use.  For  any  customer 
agency  to  do  an  adequate  job  of  "quality  control"  with  respect  to  the 
adequacy  of  the  infcrmation  shown  on  the  drawings  would  require  dupli¬ 
cating  the  contractor's  engineering  effort,  including  a  review  of  all 
facets  and  tradeoffs  concerned  with  safety,  reliability,  maintainability, 
standardization,  produceability,  performance,  logistics  support,  etc. 

As  a  matter  of  fact,  these  other  engineering  disciplines  are  being 
policed  as  suen  by  the  specific  design  and  review  organizations  which 
are  required  by  the  various  specifications  concerning  them;  i.  e.  , 
specifically --safety,  reliability,  maintainability,  standardization,  etc. 
Beyond  this  the  compliance  of  design  information  shown  on  data  to  the 
hardware  involved  is  subject  to  demonstration  it  Contract  Technical 


242 


Compliance  Inspections  (CTCI)  or  First  Article  Configuration  Inspec¬ 
tions  (FACI),  and  the  maintenance  of  the  currency  of  such  data  is 
policed  by  disciplines  such  as  "Configuration  Management.  M 

A  further  point  along  this  line  is  the  fact  that  a  contractor  is  subject 
to  the  same  limitations  when  considering  vendor  or  sub-contractor  data. 
The  prime  reason  for  going  to  a  vendor/ sub -contractor  for  design  and/or 
hardware  is  the  fact  that  the  appropriate  design  technology/ capability 
does  not  exist  in-house.  If  the  prime  contractor  had  the  capability --in 
technology  and  manpower --to  thoroughly  authenticate  the  design  com¬ 
pleteness  of  the  vendor/ subcontractor  data,  the  prime  would  probably 
do  the  work  himself  initially,  rather  than  simply  duplicate  the  work 
done  by  the  vendor. 

It  would  seem  then,  that  the  only  really  legitimate  "quality  control" 
functions  affecting  data  of  itself,  are  the  so-called  "--mundane  require¬ 
ments  such  as  the  size  of  the  title  block,  and/or  the  legibility  of 
reproductions  of  that  data  as  supplied  upon  call.  No  contractor  should 
ever  object  to  a  customer's  review  for  legibility.  We  may  challenge 
his  eyesight  on  occasion,  but  certainly  we  can  be  expected  to  furnish 
data  which  is  readable.  Further,  v  e  can  be  expected  to  furnish  data 
which  complies  with  the  mechanical  aspects  of  data  requirements,  such 
as  drawing  size,  drawing  format,  etc.  Beyond  this  there  is  no  logical 
or  reasonable  basis  for  quality  control  of  a  data  package. 

We  might  suggest  that  a  major  portion  of  the  legibility  problem  of 
data  is  generated  by  the  practice  of  the  governmental  agencies  in 
obtaining  all  the  data,  reproducing  it  for  distribution  to  prime  agencies, 
and  in  turn  reproducing  again  (sometimes  twice)  before  delivery  to  the 
eventual  user.  This  third,  fourth,  or  fifth  generation  copy  could  only 
be  made  good  if  it  started  from  the  ultimate  in  drawing  originals  - -an 
ink  tracing.  This  is  obviously  a  very  expensive  way  to  get  readable 
copy  to  the  end  user.  A  better  way  (now  being  implemented  through 
"contractor  data  bank- -'call'  contract"  procedures)  is  to  let  the  con¬ 
tractor  supply  direct  to  the  user. 

With  respect  to  the  specific  five  questions  that  your  letter  proposes, 
my  answers  are  as  follows: 

1.  I  have  no  doubt  that  the  DoD  has  some  quality  problems  in 
connection  with  the  drawings  it  acquires.  I  do  not  believe  that  the 
correction  of  these  problems,  other  than  as  they  apply  to  format  or 
legibility,  is  susceptible  to  additional  quality  control  requirements  or 
procedures. 
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2.  Our  recommendation  with  respect  to  improving  the  quality  of 
drawings  would  be  to  return  to  the  status  of  quality  requirements  that 
obtained  with  MIL-D-5481  or  MIL-D-5028.  I  firmly  believe  that  other 
facets  of  customer  control  of  the  contractor's  activities,  such  as  con¬ 
figuration  management,  safety,  reliability,  maintainability,  standard^u- 
tion,  etc.  ,  more  than  adequately  account  for  and  cover  various  facets 

of  design  completeness  and  accuracy  as  proper  tradeoffs  on  engineering 
data. 

3.  It  is  our  position  that  quality  assurance  or  inspection  personnel 
should  not  become  involved  in  checking  drawing  quality,  other  than 
possibly  for  legibility  and/or  format. 

4.  Sampling  techniques  restricted  to  the  checking  of  legibility 
and/or  format  would  be  acceptable. 

5.  It  is  our  position  that  the  only  reasonable  quality  controls  that 
can  be  imposed  on  the  adequacy  of  design  information  contained  on 
engineering  data  is  the  in-process  control  maintained  by  a  contractor 
in  the  form  of  a  checking  group  within  the  engineering  release  system. 

Best  personal  regards. 


Very  truly  yours, 


AVS:  lc 

co:  Advisory  Committee  to  OTDSP 
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20  June  i966 


Colonel  O.  C.  Griffith,  USAF 

Acting  Director,  Office  of  Technical 

Data  &  Standardization  Policy 

Office  of  the  Assistant  Secretary  of  Defense 

Room  3D232  -  Cameron  Station 

Alexandria,  Virginia  22314 

Dear  Ole: 

With  regard  to  your  letter  of  9  May  1966,  the  matter  of  the  quality 
of  drawings  has  been  carefully  considered  by  our  Engineering  Drawing 
personnel.  Their  opinions  on  this  problem  are  set  forth  in  this  letter. 

It  is  agreed  that  an  analysis  of  the  quality  of  drawings  can  be  broken 
into  two  main  divisions.  It  has  been  our  practice  to  refer  to  these 
divisions  as  the  technical  and  non-technical  quality  aspects  of  drawings. 

The  quality  of  a  drawing  (or  a  set  of  drawings)  with  respect  to  the 
technical  aspects  must  take  into  consideration  the  design  phase  in 
which  the  drawing  is  prepared. and  utilized.  As  the  design  details  become 
more  firm  and  the  life  of  the  drawing  extends  into  subsequent  design 
phases  the  drawing  approaches  a  stage  where  it  contains  a  complete 
and  adequate  set  of  engineering  requirements  which  delineates  the 
design  for  fabrication,  assembly,  inspection,  test  and  utilization  of  the 
item  described.  Prior  to  this  stage  the  degree  of  completeness  of  the 
engineering  requirements  on  the  drawing  is  commensurate  with  factors 
such  as  availability  of  firm  design  information  and  existence  of  in-plant 
channels  of  communication  by  which  requirements  are  made  known  or 
understood. 

Evaluation  of  the  non-technical  aspects  of  a  drawing  (or  set  of 
drawings)  takes  into  consideration  general  drafting  practices  for 
recording  design  information  after  it  has  been  established  and  con¬ 
sideration  of  the  suitability  of  the  drawing  for  its  intended  use.  The 
general  drafting  practices  referred  to  pertains  to  drawing  characteristics 
such  as  the  following: 

a)  drawing  sheet  format 

b)  lines,  lettering  and  symbols 

c)  types  of  drawings 

d)  drawing  package  makeup  for  an  item 

e)  legibility  and  reproducibility 
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With  respect  to  your  specific  questions  our  comments  are  as  follows. 

Question  No.  1:  We  have  no  knowledge  as  to  whether  or  not  DoD 
has  a  general  quality  problem  in  connection  with  the  drawings  it 
acquires.  To  our  knowledge,  drawings  submitted  by  .  .  .  have  been 
satisfactory.  In  general,  based  on  our  association  with  other  Aero¬ 
space  Industries,  it  is  doubted  that  major  drawing  quality  problems 
emanate  from  the  larger  Aerospace  companies.  It  is  pointed  out  that 
a  quality  problem  may  exist  when  drawings  are  ordered  prematurely 
since  such  drawings  will  not  be  complete  and  may  not  contain  firm 
engineering  requirements.  The  .  .  .  requires  that  engineering  drawings 
for  all  production  items  be  similar  in  nature  and  consent  as  drawings 
required  by  DoD  primarily  because  of  decentralized  design  and  drafting 
and  extensive  internal  use  by  one  or  more  major  functional  activities 
such  as  Manufacturing,  Quality  and  Support.  For  such  an  organization, 
uniform  drafting  practices  are  required  and  the  drawings  must  be 
complete  to  the  maximum  practical  extent. 

Cost  effectiveness  also  comes  into  the  picture  of  drawing  quality 
since  there  is  a  limit  to  the  amount  and  detail  of  technical  information 
which  may  be  recorded  on  a  drawing  and  to  the  degree  of  emphasis  to 
be  placed  on  details  and  strict  adherence  to  non -technical  quality 
aspects.  It  is  our  belief  that  it  is  not  necessarily  economically  practical 
to  expect  that  drawings  be  prepared  to  support  manufacture  by  another 
company  even  though  that  company  is  a  competent  manufacturer  in  the 
same  or  allied  industry. 

Question  No.  2:  It  is  recommended  that  analysis  and  action  be 
di  ected  to  the  known  cases  of  grossly  inadequate  engineering  data 
submittals  rather  than  tak:ng  across  the  board  actions.  The  quality 
assurance  provisions  of  MIL-D-1000  are  considered  generally  adequate. 

Question  No.  3:  The  technique  for  assuring  drawing  quality  logically 
differ  for  the  technical  and  non-technical  quality  aspects  in  our  opinion. 
Quality  assurance  for  the  technical  aspects  is  primarily  performed  by 
the  engineering  checking  function.  Further  assurance  is  obtained  through 
1)  the  drawing  release  and  change  control  systems  and  2)  the  contractor's 
hardware  quality  assurance  system.  For  the  non-technical  aspects, 
quality  assurance  is  best  performed  by  the  engineering  checking  function 
and  drawing  control  personnel  in  the  engineering  organization  who  are 
specifically  assigned  this  responsibility.  The  drawing  control  function 
provides  assurance  of  conformance  to  selected  ncn-technical  character¬ 
istics  and  to  overall  requirements  which  pertain  to  a  total  package  of 
drawings. 
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Question  No.  4:  Sampling  techniques  would  be  suitable  as  a  partial 
measure  of  conformance  to  the  non-technical  quality  aspects.  In  our 
opinion,  sampling  techniques  are  unsuitable  for  inspection  of  technical 
quality  aspects.  Sampling  techniques  at  final  data  inspection  for  non¬ 
technical  aspects  have  been  used  successfully  by  .  .  .  for  some  time. 
When-used,  sampling  has  proven  to  be  an  economical  technique  for 
assuring  quality  over  and  above  that  provided  by  the  in-process  engi¬ 
neering  checking  and  data  control  functions.  Because  each  company 
needs  to  establish  its  sampling  plans  to  fit  its  mode  of  operation, 
standard  sampling  plans  for  universal  use  are  probably  not  feasible. 

Question  No.  5:  In-procers  controls  must  be  relied  on  to  assure 
drawing  quality  with  respect  to  the  technical  quality  aspects.  These 
controls  are  inherent  in  operations  such  as  configuration  review, 
engineering  checking,  drawing  release,  hardware  quality  assurance 
and  configuration  control.  A  thorough  and  final  inspection  would  be 
impracticable  a  i  costly.  For  the  non-technical  quality  aspects, 
in-process  controls  alone  may  not  be  completely  adequate.  In  our 
opinion,  a  final  inspection  is  needed  to  determine  that: 

b)  The  engineering  data  as  a  package  is  suitable  for  the 
intended  use  for  which  it  is  being  submitted. 

b)  The  set  is  complete. 

c)  Assurance  that  the  non-technical  contractual  requirements 
for  Form  1,  2  or  3  as  specified  in  MIL-D-1000  are  met,  and 

d)  Legibility  and  reproducibility  is  adequate. 

In  conclusion,  we  strongly  support  the  Aerospace  Industries 
Association  efforts  in  establishing  a  basic  understanding  of  the  engi¬ 
neering  drawing  as  presented  in  the  AIA  presentation  "The  Engineering 
Drawing  -  Its  Purpose  and  Application".  It  is  believed  that  the  inter¬ 
change  of  viewpoints  and  information  between  AIA  and  DoD  will  provide 
a  mutual  understanding  on  the  subject  of  drawings  and  an  opportunity  to 
recognize  aieas  of  potential  drawing  quality  problems. 

We  appreciate  this  opportunity  to  emphasize  the  importance  of 
drawing  quality  and  to  provide  our  comments  on  this  subject.  We  trust 
that  the  above  comments  will  aid  your  efforts  in  determining  the  adequacy 
of  data. 

Best  regards. 

Sincerely, 


cc:  Advisory  Committee  to  ORDSP 
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June  16,  1966 


Col.  O.  C.  Griffith,  Acting  Director 

Office  of  Technical  Data  and  Standardization  Policy 

Office  of  the  Assistant  Secretary  of  Defense  (I&L) 

The  Pentagon 
Washington,  D.  C.  20301 

Dear  Ole: 

With  respect  to  your  letter  to  me  posing  questions  with  respect  to 
the  assurance  of  the  quality  of  drawings  furnished  to  government  agen¬ 
cies  by  contractors,  I  have  the  following  comments  which  were  devel¬ 
oped  from  comments  by  cognizant  personnel  in  .  .  .  . 

1.  We  agree  that  DoD  has  a  general  quality  problem  in  connection 
with  the  drawings  it  acquires.  This  opinion  is  based  on  the  microfilm 
of  other  companies'  data  which  we  have  received  from  the  government 
to  manufacture  equipment.  However,  I  am  sure  that  if  you  ask  each 
company  to  comment  on  its  quality,  you  will  be  assured  that  that  company 
submits  drawings  of  the  highest  quality  to  the  government.  My  personal 
opinion  is  that  somewhere  in  between  lies  the  truth. 

2.  It  is  our  belief  that  the  requirements  in  the  specification 
governing  drawing  content,  format  and  legibility  are  entirely  adequate. 

It  is  the  enforcement  of  these  requirements  that  is  not  consistent.  In 
addition,  there  does  exist  the  problem  of  the  differentiation  between 
"technical"  content  and  "format"  and  it  is  our  experience  that  "format" 
rather  than  "technical"  content  has  taken  primary  importance.  There¬ 
fore,  in  order  to  differentiate  it  is  recommended  that  "format"  and 
"technical"  will  have  to  be  more  definitively  defined. 

3.  Most  companies  have  drawing  checkers  who  check  drawings  for 
"format"  and  "technical"  adequacy.  However,  quality  assurance  or 
inspection  oersonnel{as  distinct  from  drawing  checkers)  are  usually 
qualified  to  inspect  drawings  for  format  conformance  only  because  they 
are  not  normally  equipped  to  pass  on  the  technical  adequacy  of  drawings, 
therefore  checking  outside  the  drafting  room  should  be  limited  to  formal 
requirements. 

A  closer  liaison  between  the  inspector  and  contractor  personnel 
early  in  the  contract  would  help  eliminate  many  of  the  problems  dis¬ 
covered  at  the  end  of  the  contract.  Differences  of  opinion  in  many 
instances,  and  they  are  just  that,  could  be  resolved  early  in  the  contract 
and  uniform  ground  rules  would  thereby  be  established  sufficiently  in 
advance  to  resolve  many  of  tin*  problems  which  are  minor. 
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4.  Sampling  techniques  will  not  assure  drawing  adequacy.  Legi¬ 
bility  of  a  print  will  provide  assurance  that  another  print  made  from 
the  same  drawing  will  be  legible,  depending  of  course  on  the  number 
of  times  used.  The  fact  that  the  drawing  is  correct,  however,  will 
not  provide  assurance  that  another  drawing  is  also  correct.  Both 
drawings  being  dissimilar  items  must  be  inspected  for  all  attributes. 

I  must  admit  that  this  opinion  is  a  consensus  and  that  there  is  not 
unanimous  agreement  in  .  .  .  on  this  point. 

5.  In  process  controls  can  be  relied  as  to  assure  conformance  of 
drawings  to  all  requirements  except  legibility.  Legibility  degrades 
with  drawing  use,  all  other  attributes  do  not  change.  In  this  connection 
we  would  like  to  comment  that  the  state  of  the  microfilm  art  is  not  yet 
adequate  to  produce  the  consistency  legible  fourth  generation  microfilm 
required  by  MIL-M-9868. 

Another  point  worth  mentioning  is  the  fact  that  each  military 
activity  has  its  own  opinion,  or  sometimes  no  opinion  at  all,  as  to  what 
constitutes  a  satisfactory  data  check  either  in  content,  format  or 
acceptability.  There  should  be  an  educational  program  instituted  which 
would  create  a  greater  dependability  as  to  the  requirements  of  a 
satisfactory  data  package. 

There  still  remains  the  problem  of  adequately  defining  drawing 
requirements  and  the  kinds  of  drawings,  e.  g.  ,  the  continuing  dispute 
over  MIL -D -1000.  If  you  have  difficulty  in  defining  the  requirement, 
how  do  you  know  what  you  are  to  inspect. 


Very  truly  yours. 


